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Executive Summary . ' ' . 




■". The following report describes an evaluation of the utility and 

'.»■,' 

^ perceived effectiveness' of a spedlfic, structured analysis and design techjjlque 

■ ■ ' ■ " .,»,>-. ' * 

as ari^ald In managing a complex, military training environment . • In this study 

SofXech, Inc., a Boston-based computer software firm, applied their Structured 

* ATjalybls and Design Technique (SADT^ ) to the problems confronting the 

/. "jl^^ and Doctrine . Command (TRADOC) In managing rapidly occurring 

. Innovations In training. - , . . 

(^Jk^^^^^y^ These changes Include* the self-pacing of a, large number of training 

cfturses, evaluating training effectiveness both, in terms of the individual 

soldier' and .the unit, and the development o£ management strategies to -improve 

TRADOC' s ability '^o allocate resources, establish accountability, Inxprove . 

^ eaclAngju quality, and tcy"^ a*better match between training content and. 

^ objectives. ^\ • 

If "use:,of tl^b^pXti(^ular;.analyais and planning mettiodology signit- - 
ica^tly Improved ^XRADpC' f^bllxty to manage proposed changes in training , more 
widespread u.^e of ^SADT in tJgL DoD training; environment would be warranted. 
»»Thus, the^roject included an . Independent evaluation co^mnissloned by the' ^ 
Advanced Research Projects Agen/y (ARPA) and conducted, by The instructional 
Systems Llaboratory at The University of Texas at Austin. The ^evaluation 
staff was responsible solely to ARPA for evaluative procedures^, ^fesvilts, 
and conclusions. The go^als of this evaluation were to assess the utility 
and effectiveness of SADT in' ♦a training environment. This document constitutes 

the final report of this evaluation. / . 

^- . f ■ - . ■ 

Overview of SADT , ' c ' ■ * s 

V^'"^ * ' . - ■ 

SADT analyzes a problem through building a m^el or representation of 

' / ■ 

the problem on paper which is topj-down, hierarchic, modular and/^ruc'tured . 



SADT consists of two major components, graphic Rotation and a disciplined , . 

process which specifies how individuals in clearly defined roles/ are to, , J 

interact to produce a. model of the training program. The benefits of ^ 
applying this technique to a complex system problem derive from^the mocW^l 
produced. This model' serves as documetijtation of, the best thiijlclng of experts 
as to the status 
comraenter cycle- 



of t^e system and ey(i^lves through -ai^i iterati|[e author*- 



Project Tasks and Evaluation Phases 
» SofTech was to perform their work in 4 tasks. Task 1/ was a "start up" 

activity in which SofTech analys ts^ produced an'ovef^iew model of the training- ^ 
and evaluation system. In Task 2 SADT analysts from SofTech- and TRADOC : 
jointly produced an SADT model of a tank weapons system. In Task 3 TRADOC 
and SADT analysts .used Tasks 1 and 2 models as starting-points to produce 
a fully elaborated model of Army training and evaluation as it should bcj. 
conducted. Task 4 was to consist of a plan far implementing the training^ 
and evaluation system conceptualized in Task 3, but was , subsequently altered i 
and did not fall within the ^timeframe of this evaluation,' • | 

The evaluation was conducted in three 'phases . A primary source of 
data used in all. three phases was the expressed opinions and attitudes of ' ^ 
participating, TRADOC personnel towards SADT. The purpose of the Phase 1 
.evaluative phas6 was to determine initial attitudes towards SADT and t;o 
measure the kinds of activities occurring in the project. The Phase 2 evaluation 
reported attitudes towards SADT after development of the major SADT models 
were completed. The Phase 3 evaluation followed Phase 2 by 6 weeks and was 
concerned with ex/imining discrepancies between. Phase 1 and Phase 2 results 

along with obtaining summary impressions and conclusions |bout SADT from 

f . ■: ' "... • ■ \\ . • ■ ■ 

! participating TRADOC personnel. 



^ ■ Extensive data were qolletted *eariy in the projec^ to determine if the .* 

-SADT propess had been implemented as" sp^cTif ied -in the cpntr]act ' proposal: Theae* 
^^ta suggested that the participation of TRADOC officers in the project had 
been apptoximately half that originally expected. The number of officers' " 
trained to create SADT diagrams ^^ias^les.s than expected throughout the project 
i'nd the failure to fully implemenr this aspect* of the ' SADT proces.s at TE(ADOC 

may have lowered the impact the technique had on the client organization. 

.• . ' - ' * . ' ■ ■ ■ . ■ J 

Participating ''^^RAboC officers were' polled about halfway in the project 

- , \ ' ■ . ■ * V ■ ■ . 

concerning their, impressions of SADT and the progress of the project thus far. 

Th^ general opinion-expressed was that jwhile the technique appeared to be a 

^ '. ■ ' * < ■ ■ ■ ■ . 

good one- in a general, cont^t-^free sense, most officers believed it was too 

('.., .'.v ■ ' . 

early to conclude whether SADT would ultimately affect the training concepts' 

held by TRADOC, Some officers were skeptical of this project having any 

impact on TRADOC or ^jt^he status of training. , * , . 

Participating TRADOC ^of fleers were polled again following the completion 

of Task 3. It was found from comparing these two samplings that attitudes 

towards SADT had become more positive following the completion of/Task 3. 

At this point participating officers. had gained confidence dLn the ability, 

of the technique to focus attention on relevant issues and elucidate the 

interrelationships of various components of the training system. 'These 

officers were particularly pleased with the Task 3 model, which * appear s to 

have documented a level of understanding and insight into . th-e ^^raining system 

which had not been achieved or at least-so viv,idly portrayed previously. ;A \ 

follow-up interview and questionnaire six weeks later indicated that these 

expressed opinions remained stable, 
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SjUtmnaty qnd Conclusion^ 

^ ^ ■ ! ■ ■ 

•i V Articulation of Army training and evaluation reptesented by tl^e 
Task 3 model represented' the> most d^^re^t /tangible benefit of this project 
to^-TRAbOcV' This model Ihtegt^ted the/ prlvloup^ independently conceptualized 
ateas of training, £val|jation, and systm development in an^ understandable 
ani ptacdical ;manner. From this documentation, the actual work of imple- 
'menting; the^ new c(3hceptudlization of training can begin. 

- It does not appear that this application Of SADT created new know-, 
:le<ge about spieiiffic- aspects of the training system but rather that it 
. integr^ite^/and communicated what, was known about training in a useful and; 
cohetenrt: way. the>; SADT ^dlag^ms and procedures provide a. mechanism for 
documenting the bes^thinking in a client organization and iqt:et;relatlng. ^ 
important concepts in a clear, and practical manner. It was Concluded that 
further application^ gf SAD.T such as the one described in this report, would 
be worthwhile! . . ' J . ^ 
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* ^ I. Project Overview * ' 

/ ' • ■ . ' '* ■ ' ■ 

This document describees the evaluation of a project^ supported 
jointly b^' the United States Army Trairiing and Doctrine Command (TRADOC) 
and the Defense Advanced Research Projects Agency (ARPA) under the 
contract title "Demonstration and Evaluation of an Advanced; Syitema Analysis 
Technique in. Modeling a DoD Training Environment" (ARPA Order No. 3230, j 
Contract No.: MDA 903-76-,C-0249) . ** / 

In this study SofTech, Inc., a Boatoh-base^ computer software firm, 

applied the Structured An'alysis and Design technique (SADT>r ) to the - . 

problems frtmf routing a training command headquarters i,n managing rapidly ' 

occurring innovations In training methods. These changes include the self- J 

pacing of a large number of A/tmy courses, the initiation of a major project 

to evaluate training effectiveness both in terms of the individual soldier 

and th^^unit, \and the development' of management strategies to improv'e TRADOC^ 

ability to allocate resources to training sites, establish better accounta- 

' ' , .If} 

bility of resources expended, Improve the management of teaching quality, 

measure competence objectively and quantitatively j and to achieves better 

match between training content and real wotld job objectives. 

If "this particular analysis and planning methodology signflcantly 

. : : • ^ ' ■ ■ ; ' , , 

*The authors .would like to thank SofTech, Inc. for their cooperation 
and assistance in gathering data'for this report , especially Reuben Jones, 
project manag^er; Steve Lipka, Stan Smith and Boyd Matchers, SADT authors; as 
w^ell as many officers at TRADOC who freely gave of their valuable time for 
^interviews and questionnaire responding. Without the cooperation of both^^ 
agencies this report could not have beep written, 

> **Evaluation reports were generated throughout this project. These 
* interim reports were written for a limited audience and were not disseminated 
Although not available, these reports are cited ±n the following text to pro- 
vide background and clarity to ongoing discussions. The substantive results- 
and conclusions of these smaller, sequential project summaries are contained 
in this report. . ^ . 



improved TRADOC's ability to accomplish thpse* objectives, more wide-8{>read.-<'.'' \. " 
use of SADT througKout the DoD tr.aining establishment,, and possiblyf other V 
parts of .DoD Would Be warranted,' Because of the, rapid pace with which . 
trj^lning' innovatibns are planned at TRADOC, this environment provided ^ 
good test bed in which to assess the applicabil:j,ty of SADT- to analysis and 
planning prpblems encountered in , other DoD' training envirpnments: ' 

.... . . / • ' ' ' / ^ 

Thus, this project included an evaluation -effort to assess the utility 

.and ef:fectiveness of the Structured Analysis and Design "technique (SADT) . 

In a trainiilg. environment and pn evaluatioif of *the impaicit of the project 

• * ■ ' / ■ - - • 

Gin TRADOC*^ ability to identify -and understand the types ^pf*^changes required 
in Array training atid to develop a practical plan for carryin^^out those 
changes. This document constitutes the final report of this evaluation. 



t , . \ II. Overview of SADT \ y 

* ■ * ■ ^ " • fc ■-.■>-• . ^ . ■ , * . 

SADT is a systematic approach: to understanding and so IV iVig complex 
system problems, i'ncluding^^plTm^^ functional 
analysis, and system design. Under development by SofTech since 19f0, the 
technique grew out^ of earlier work on software engi^iieering at the Massa- 
chusetLs Institute of Technology. SofTech has applied the technique. ' 
to a wide range of planning, analysis, and ^lesign problems, involving 
people, machines, software, hardware, databa)^|^, communications networks, 
procedures and finances. ' ' 

The underlying philosophy of SADT is that the human mind is capabl 
of understanding any amounts* of complexity, as long ag it is presented in 
, small, accessible ctufcks that are linki^d together to make a whfble. This 
structured decomposition o^rientation^ to understanding complex problems has 
become a standard tool of computer software !developmei>t specialists but\' 
would apjjear to be> applicable to any f ield where there is a need to eluci- 
date the. relationships between parts and wholes" in complex systems- or 

t ■ <. . . 

programs . . ' 

/ ' • . , 

■ / 

SADT analyzes a p'rO^blem through building a model pr representation 
of the problem on paper which is top-doVn, modular , hierarchic , and . , . 
structured. SADT consists of two major components, the graphic notation 
of this particular modeling methodology and a disciplined .process which 

'specifies how individuals in clearly defined roles are to. Interact^ to produ 
tiip model . Thus , ^ SADT consis ts of both a^raphic language and a well- 
defined discipline or« process which spells out * tfe procedures to be followe 
in. obtaining a structured decomposition of the problem at hand. These will 

*be discussed in turn below. , ., s ' / 



Qi^aphlc Convention s . The SADT graphic ianguagfe .provides a limited 
set of 'constructs from which analysts- and designers , can compos^ orderly 
structures of -any required size. .The notation Is compjosed of, boxes and 
arrows . Bpxes represent parts of p. whqle, atroifiM r^pte^ent interfaces between 
part^** Diagratas represent wholes and, are composed ai: Jbokes, arrows^ n£^tura\ . 
language .names, and certain other ho'tation. The-same graph^lcs are appjk^|ble 

0 ■ ' ' 

^ both activities' and dat5. ' < , > ' ' * ' 

An, SADT model is an *oi/ganized sequence o'f diagrams, each witH* 
concise supporting text. A high-^ level overview* diagram represents the wwUte 
subject. Each lower-level diagram connects exactly Intlo higher-level portions 
of the model, thi^s preserying the logical relationship of each component 
tff) the total system. Thus, program detail is introduced gradually so;- that 
substantive detail is Integrated into the whole without obscuring the-, 
overall Int'ei^t or "big picture". Figure. 1 provides a conceptual Illustration, 
of the thodellng process. Figure 2 contains an actual SADT diagram ..produced . 
during this project. ^ - ' . - 

' An SADT model is a graphic representation of the hleralrchic J^trUc- 
ture of a system, decomposed with a definite pur4)oae\in mind. A model is 
structured so that it gradually -exposes'^ detail, but its d^pth is bounded, 
by the restriction of its vantage point ;l and . Its content is bounde^d by Its 
v*iewpoln^. The priorities dictated by its purpose determine the layeifing 
of the top^pwn- decomposition. Parallel model^ can accomodate both multiple 
viewpoints and various /stages of system implementation. • 

'. Jhe arrow structure ou an SADT diagram representsJ a constraint 

' /. ■ f . , ' ■ '"'■*•..■.■ 

relationship among the boxes. It d&es not necessarily represent flow of 

control or sequence,':: as for example, on a flowch*krt for a computer program. 
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More or tailed 



Figure I. Conceptual illustration '^^f the modeling 
proce!^. 



Constraint arrows show necessary conditions imposed an activity. 

Most arrows reprot^ent Interfaces between boxes, whether in the 
same or different models. Some arrows represent non-interface Interlocking 
between models. The interface structure, particularly', passes through 
picVQVwi levcvla of diagrams, creating a web that integrates all parts^^of the 

[Bm's ^environmental interfaces with 
further documentation of the mechanics o f ^the technique 

©(Publication number 9022-78R, 



decomposition and shows the whole system 
the topmost box- ^ 
can be found in An JdlLL^^l*^- ^ l^^l^tc^^ 



SofTech, Inc., A60 Totten Pond Road , Waltham, Massachusetts, 02154) 



The SADT Vfj^Sl^^.* Complex problem analysis requires cooperative 

. a r.-lear definition 



teamwork f rom many I ad i v I (hia 1 s . SAOT attempts to provide 
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■Figure 2, SADT diagram produced during this project. 
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of the kinds of i'riter^ctions whick shmilcl occur between the personnel 
involved by establishing titles aill fiftictiotis of appropriate roles . These 
appear in Table 1. SADT proc^s^, , in Hn^ch these role^s^^^ii^ provides 

continuous documentation, and reeulkK^li:^^^^^^! ^^^view of the work prc^huced. 
In this way decisions can be ^ee|^ in context and can be challenged v^ile 
alternatives are stlil viable. \ ^ 
Throughout a project , 'draft versions of diagrams in evolving mode^^s 

are distributed to project members for review. Commenters 'make their 

* ■ 

suggestions in writing directly on copies of the diagrams. Written 
record^, of decisions and alternatives are retained as thejy^ unfold . As 
changes and corrections are made, all versions are entered in the pr.oject 
files. A project librarian provides filing, cflstrlbutlon ar)d record ' keeping 
support. This process documents all decisions ari^^^tirc-^sons why decisions 



are made. When coramenters and authors reach an understanding, the ^ork is 
reviewed by a committee o E ' senior technical and. management, per9<^nnel . During 
the process, Incorrect or unacceptable results may be identified #arly, 8o 
that oversightH or errors can he dctocted before they cause major disruptions. 
Since everything Lh record, futurp enhancement and system maintenance 
can reloronce previous analysis and de'sign decisions. A list of ryles and' 
functions used in ihc SADT process appears in Table 1. 

Because documentation is produced as the model evolves,, the 
status ot the .project is visible to all interested parties. ManaBemont can 
study i.he ,t^«quiremeuts (or the design) in a "top-down" manner, beginning 
with an Y^cvvicu and continuing to any relt^vant level of de.>rt-H-^Altliough 
prescntatrmT^; to upper management usually follow standard \summary and 
w,i(k-tln-ougli mctiiods. oV(mi r.vixioi executive.-, can become flvuiiii-^: tlio 
SADl' l.uiguage. 




Table 1 . • 
Titles ^nd Functions in the SADt Process 



Title 



Aut 





r 



, Conunenters 



Technical Committee 



F'Xperts 



Mon i tor 



Proioc:t LibrarLnn 



t: rue Liu* 



Funct ion 



' Personnel who sjtudy requirements and 
constraints, analyze' the system functions 
and represent them by a model based on ^ 
SADT diagrams." ' ^ * 

Individuals who must review and comment 
in writing on the SADT diagrams produced 
ty the authors. -0 ' 

A group of senior technical personnel 
"^agsigned to review the analysis; at 

every major level of decomposition. They 
resolve technical questiotis or disagree- 
ments or ^recommend a decisiorr to the' 
project management. . " . ^ 

Persons from whom authors obtain special'- - 
ized information about -functional require- 
ments and certain constraints- by means 
of interviews . ' ' / 

Personnel who redd SADT diagrams for infor- 
mation but are not expected to inake written 
comments. - 

A person fluent io ^ADT who assists and 
advises project personnej. in the use and 
application of SADT. 

Aspersion assigned the responsibility of 
maintaining a centrali'zed file of all. 
project diagrams and 'associated documents, 
making copies, d.ls tr ibuting reader kits, 
keeping reader kits, keeping records, etc. 

The member of the project who has the 
final technical resR^nsibil 1 ty for carrying 
out the system anaVysLs and design. 

A poL\;^on fluent" iuNvADT who trains Authors 
^md Commeuters^' who al:e using SADT for the 
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Implementing the .Approach . How SADT ; Is Implemented varies according, 
to- organlzatioa. needs and the' kinds of systems under consideration. There 
Is no set 'pattern^-ain©ng 'different organisations for tne contents of a 
problem analysis. In each case> the needs of the client ofganizati^on" must . 
be accomodated,. Beoause local needs' are divjsrse, implemetntation of SADT is 
a "learn by doing'! experience in which pro j ecft^ personnel acquire ways of- 
understanding the generic nature "o f systems. 

While little previo^js evaluative data ekists on SADT , . users' r.eport* 

that It is .a communications vehicle which focuses attention ou well-defined 

". ' ■ ^ 

topics, that it increases management control' thtough visibiliiiy and standard- 

<^ , ■ , \ ' . 

- ^ / _ 

ization, that it creates a systematic work breakdown structure for project - 

< . ^ ' ■■ *" ' ■ 

teams, and t^at it minimizes errors through disciplined flexibility (see 

RosR, D.T. and Schomau, K.E. Sttuctured analysis for requirement s definition , 



IEEE Transactions on Software Enginee ring, Vol. SE-3 1^1), January-^ 1977, pfi^. 
6-15) . > ^ 
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• >IIi. Overview})f the U.S. Army Trai^in'g' 
'j) 'i : aXJd^Doctrine/comraaigtd (TRADOCy* 




The U.S. Army Training, and' fioctrine Command (TRADQC) was created 

"iri^lQTI^.v With/^an annual budget approaching one billib/ dollars and a total 

program that Include? the emplpyment of ^ approximately 12,000 officers/ ; 

50;000 enUsted men an'd' 40,000 civilians , TRADQC" is ^^charged with the responsi- 

bUltHof. establishing and' maintaining. Army concepts, principles, and policies 

Cddctrineij ►and forjthe conduct of all Army training. ^ , 

JRAtofC consists- essentially of two major components for meeting ' ' 

ytnese responsibilities: a combat developments program and ' a training program. 

. The combat developments program consis'ts of three major activities *(1) the ^ 

-determination of requirements; ap^ capabilities of weapon systems* and 
• ' • . • ■ . ■ " ■ ' * . ' ' . " . / ■ 

equipment (including non-comh"at equipment) ; (2) the devejilopment of optimal , ' 

. organizational policy (e.g.", determining the formal organizat;^on of tank 
battalions and the integration of those forces with mechanized rifl^ com- 
panies, maintenance crews',^ etc. ) ; and- (3) thei development of tactics and 
tzt^chniquGS by which the acquired weapons and equipment will be deployed - 
by the various organizational' groupings of troops on <the modern battlefield. 

The training br^lnch of TRADOC consists of the tr^itional* Army 
train Lag structure described below, and a training development component ^ 
which has achieyed visibility only vs^i thin the last two years. Each 
be discussed tjjtlow. ^ » 



*ThLs deticript ion of TRADOC is based on tlie pi'occcdingi^ of various 
Commander's Contcrcnccs and Technology Symposiums sponsored by TRADOC in 
1975 and 1976. Wlrilc this overvLew may noW be sli^Uitly dated, it illustrates 
('t)us Lderatlons which ultimately led to TRADOC's participation in this project. 



While the training developments component of -TRADOG represents 
new concept, the training sSructjire within the Ari^ly is wall established. 

- ^ . ) , • ^ \ ) ■ ',\^ 

For enlfsted men, the , program consists of the well-'kno^m basic training 

■ a 

and advanced individual training (AIT) which is conducted at several 

■■'^ .... \ 

insCallations across the country. There are 21 .training schools such 
as^the U.S. Array Infantry School^ at F^, Benning, Georgia, the U.S. Army 
Armory School at Ft. Knox, Kentucky, the U.S. Army F^cld Artillery School 
at,. Ft. Sill, Oklahoma, the U.S. Army Transportation School at Ft. Eustis, 
Vir<:inia, and the U.S. Army Engineer Training Genter at Ft Leonard Wood , 

Missouri. There is a continuing education program for noncommissioned 

/• ' ' . " ' ' 

officers (which, like Alt,. Is usually conducted through the training school) 

> ■ ■ ■ ■ '( ' ' ■ ' 

♦ Lo'keep soldiers current on developments in equipment nnd dpctrinje. 

There is also basic training, intensive advanced in^dividual 

r . ' : ■ . ■ ■ 

training and continuing education programs for officers which contain 
3Decial emphasis on such arejO^ as leadership, management, and tactics. 
T^or higher ranking of (: Leers (usually majors) there is the Gommand and 
General Staff GoMago at Ft. Levenworth, K'^insas, and at a still higher 

level, there is the Army War Gollege. Additionally; che entire ROTO 

^ ^ \ ' ' / 

structure and Officer Candidate School fall within TRADOC'^ responsibilities 

In .addition to these training facilities, TRADOG contains many 

^^snoagencies such as the Combat Arms Training Board (GATB) , the Training 

Aid Development and Requirements Agency (TRADER) , the Combat Development 

L'xperimentat iOjH Gommand '(.GDEG) , the TRADOG Combined Arms Test ^g'ency ' 

(TCATA) and the TRADOG Systems Analysis Agency (TRASANA) , Thu:;, .mana)>e- 

mcnt "responsibility for the Various aspects of training and evaluation 

' \is traditionally bciau diffused throughout the tralnin\^ system. 
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While this brief survey 6f training commands and agencies;- is not 
exhaustive, it indicates the brondth of TRADOC's imIvitmI li>nal m;ui.iK4MiuMil 
responsibility. Thus, TRADOC^^presT^uts pe theworUrs Largest 

training efgency, respons/ble for teachitlg a comprehensive grange of skilla 
r""and competencies across a ^wide range of geographic locales under the 
control of a wide ,rang^ of subagencies within^ TRADOC . 
Innovations in Ayiyiy Training 

. - ; 1 1 ■ - , . 

Until reciently, innovation had not been a high priority In Army 
training and evaluation. Traditionally, job training had been conducted 
in -the Army schools using q/ plat form 4fecture format. Performance usually 
was .evaluated by paper and pencil tests. For each military occupation 
specialty (MOS) there was an AIT course and progressive suppl^&mentafry 
t raining courses. There was an associated test for each skill level .of ^ 

^ach MOS, usually consisting of about 100 mu.ltlple clioice questions. 

• • • ■ . . ^ . .. . ^ ^ ^ , 

Initial analyses of training effectiveness in the early 1970. s 

■ ■ ' (-■ V . . - 

suggested that the actual jaerf ormance of soldiers in the field did not 
^^mccl criteria establ^hcd by outside contractors and the combat develop- 
L'-nts branch of TRADOC . A brief example illustrates this point. While 
the Qxample presented is hypothetical, it is based on the results of 
sovi^ral actual analyses of specific weapons systems. / 

Figure 3 indicates the probability of hitting a moVing target * 
uith a particular weapon as a function of the distance from the target 
in meters. Line A Indicates the actual performance of a group of soldiers 
using, this weapon. Line B indicates the capability of the weapon as 
determinod by th6> Army Materiel Systems Analysis Agency '(A^JSAA) . Line C 
/ represents what a group qf (^.ombat officers believed was the capability 
of their soldiers of using the weapon. 
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Figure 3 data suggest , that the ability of 'the soldier to use the \' 
vi^eapon in the field did not, match the weapon's capabili ty , (discrepancy 
between lines A and 'B) . Further, what commanding officers believed io-he 
the capability of the weapon w^s not veridical with actual performance ^ 
(discrepancy between lines A and C) . Thus, the decie^ions an officer made 
on the battlefield relative to/ this weapon would not match his troops^ 
ability to deploy it nor the weapon''s actual capability.- \ ^ 

Based on these resul ts the length of training • ^or t^jris.weS'pon ^ 
was doubled and ci simulator developed to provide more practice opportunities-. 
This' increased proficiency considerably • In experimenting|with the training 
for this weapon it was found that, saldlers had ' diff iculty es^iniating the 
roxige of the target through the weapon's complex sight. To remedy this i't 
was decided to use the weapons In pairs and firing In.sequence* Thus, one 
soldier would 'estimate range, aim, and fire. A second soldier would then 
use the first soldier's range estimate and his observation of where the 
*first roUnd hit to obtain a better range estimate for a second f-irlng on . 
the target. The perfornuuico of pairs of soldi'eT:s ua'lfig this px^oc5|||||^e . " ' \ 
is indicated by. Line D on Figure^ ''h This level of perfovnance exceeds 
what was originally cortsidered to be the- .Cc^pab^^ -:ie weapon. Training' 

nnalyRls iitutlies sucli as this wore seminal 1^ brlagxnr about a concern 'for 
training developnic;Ur, within l^RADOC - 

Another 'factor motivat Ing the soarclji Tor improved training methods 
was budr>^^tary cohstraluts. As TRADOC was becoming aware of some inadequacies. 
In training, they were faced wttii tlie task of i^mprov t.n^^ the quality of^ 
'training witii a^d,ecreascd tralat;^g budget. This forced the realization that 
tho^tradltionnl labor iifteiislve methods -Af training (Mnp.loyed by TRADOC 



V. ■ ■ ■ , J . , 

(platform lecturers, low instructor/istudent ratios,* etc) were not a., x j 
cost efficient approach, for future Army trainiog. Furthermore, rapst ♦ . 

soldiers spend only 2"ar, 3 months out -of a three-year tour of duty at'\ 



training school. The man. hours, lost while a soldier^ Is away fr6hi his 
wnit of assignment attending school and the associated cost^ contribute " 
to the cost of ti'.aditional Army training. These considerations have 
led TRADDC/ to' make training' available to the soldier in his^unit ;of 

assignment This was seen as no small task in that almost all training 

.■ I ■ ♦ .' ■ 

resources .were located at the schools and training centerte. 

■ - • . . ■ \ - 

Given' these consideratioris , TRADOC formed an instructional tech- 
nolqgy study group whose membership inclufcd representatives c?£ industry, 
igovi^rnment , . academia, and the military. After studying the Aritiy - train j^ng 
5;ystem this group presen ted its findings and reconmiendations, to the leader,-^ 
ship of TRADOC. . While thpse findings are too lengthy .to ipres^nt" here 
several key issues that emerged are summarized below to convey the flayo^ 
of what was re parted' by this group-. 

1/ The Army^s i nr.tnu'l UMial strati^g Los wort^ nol as cost- 
eft' Lc lent: as sc\me more, modepn instructional techno l^ogi es , 
. ' 2. School personnel lacked the understanding, motivation, 

:ukI "rdc i 1 I s t o oinp 1 oy mode rn. Ins t: ru'ctb^uia 1 t c c hno \ \ es . 1^ 
\. I'. ff ecu lye traluiny', pro^;iMni evaluation was not occurring 

ill ihe\Anuy lrainiu}\ syf. tern. 
A ♦ (>r* I ttM' lon-rtMtn'LUiced training and evaluation was not ' . 

eviilent in t hc^ Army M'ainiu^', i;yr>tem» 
^ S. V(M'v litt le sell p;uMn^; whfi IVolug tuuployt^d in. the training 
"v . . , s VM t.tMn. 



^v-W. . . ' ; ^ .... ^ ,,, ;y .; . . 

- • . ' 6. Dlslncfentives fq^r the Infcrotluc tlon of training 

- » Innovati'jon^ . abounded in the training system/ i 

' - . • ^ 7v/.;There was rib pVoponency within TRADOC diawted to the \ 
• r , ■ management bf training effectiveness. 

8. The organizational and management procegees^ of TRADOC^ 
, .were f ractionalized , or diffused to the e5ct>eht. that' ' 
■r- training program development., implementation, and 

. evaluat |p|i-were. i^^ - . . , . yV ;. 

•.V From their study',, this - consulting group recommended that TRADOC 
consolidate their tri^iaing resource management pro^esaqg^ form .a training 
management " instttute to orient and educate the various managets . of trairiT-; / 
ing in modern educational technology consider adoptitfg a systems- 
approach to training based on J oh' -performance data arid cr iterion-ref er- 
enced evaluation, and provide incentives to school -and unit command.^rs 
for iniorporat^y^ng advanced instructional techniques and establishirig 
pilot projeC'ts towards these g6alH as soon as possible^* . ' ^ .'y - - 

isince Its LiiceptLori in 1973, TRADQC has been active in its 

' ' : ■ V;'' : \ '. : i ' ♦ 

a'ttompts to modernize Army trniriing, primarily through its training. 
deyc^-LoiVmerits branch. Ma j or . proj ccts have incl'ucicd the development of 
■packaged LraLning extensLon courses (Tl'X's) which jL'.an^ be sekt to the » 
^uiiitis thus making It ciisior tor the MoLdier to learn on the. Job. Most 
oL tihese cuurses uLllizt^ an audio-vLsuali cassette machLna that presents 
a lesson both verbally and throu^\h v i LUial althi , and allous^ for testing 
and tetMlhack throughout the le.sson. Some machine^i" even allow .simulations : 

^ *'l1ie ator.ement Lcviied a^t: are not^ ,Luteu'.hNl to b(^ exhaust lv<^ . 

ol » heMurLuenct^s Lead ing tiV an increased concern for the^ status of training 
Lu Llie.Army but rather exemplary of the considerations whU-h led to chang*^. 
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of the task being taught. Evldence^'lndlcates that the^ self-paced TEC 
courses are at least as effective as traditional Army training inc^thods. 
At the end of FY 77, 1044 of the SOOO.^ourares which had been proposed 
had been developed. By 1979 the Army expects to have 5 milllfon copies 
of the TEC, lessons available for s^ive and reserve troops. 

Primarily through the efforts of the recently formed Training 
Management Institute, Skill Qualification Tests' (SQTs) have been developed 
to replace the old pencil a^d pdper MOS tests. These are criterion-refer- 
enced performance teists based on job task Analysis for each MOS. TRADOC 
Is well on its way toward developing SQTs for each skill level- of each 
MOS and towards the development of a prototype task description data base.. 

Another major effort has b^en devoted to the development of 
training and evaluation of collectives (gi^oups) . The Army Training and 
Evaliiation Program (ARTEP) is an initial effort in this direction. ARTEPs 
are manuals containing "how to" instructions for unit commanders on 

^' ■ •. % ■ ■ 

conducting collective training and evaluation. They describe the dues 
and conditions under which a partii^illar task may occur as patt of some 
large def i^^d^mission (e.g. , deliberate daylight attack" or "night with- 
drawal" of a rifle company).^ The ftRTEP manual also specifies the training 
and evaluation standards for judging th^ performance of a group. 

Another major TRADOG effoFE^has been the developiftent of l- .-^^tors 
to better train soldiers in the use of various weapons and e<^uipment. One 

y 

example is the LASER Simulator for the M16 rifle, ^which emits an eye-safe 
laser beam when fired, rather than a' live round. Sensors on targets 
provide immediate fecdbaA 'on performance. It has been found that training 
with LASER simulators is at least as effective as traditional training ^ * 

. - 27 
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with live ammunition at a fraction of the cost, ^ 

^ In collective training, two-sided free play battlefiold enpagrment 

I 

simulators have been developed. One such simulation, REAL-TRAIN, requires 

that each soldier of two opposing forces wear specially tre&ted numbers 

/ .' '• . / 

on their person. Each soldier's rifle is. equipped with a special sighting 
lens. Upon, di'ghtitig an Opponent, a soldier shouts the opponent's number 

to a referee who records a "kill" and makes sure the victim retires from 

i • , , • 

the game. The Multiple Integrated Laser Engagement System (MILES) incorp- 
orates the LAS^IJ simulator in a collective two-s^.ded free play exercise. 
Each soldier weirs a belt with lateer sensors. A loud tone indicates to 



the soldier that he is the victim' Df a hit. If hitV the soldier must go 
to a referee who^ can turn off the tone with a key. A softer tone indicates 
to the s'ibldler that he was the .subject of a near miss. MILES and REALTRAIN 
simulators have' also been developed for tank versus tank and soldier versus 
tank simulators. Further developments have been aimed towards integrating 
ARTEPs w|th REALTRAIN and/or MILES to obtain the spedificity of the SQT 
evaluation of individuals at the collective training level. 

This surv'ey of the TRADOC efforts to modernize Army .training I 
Is far from compH^te but is indicative of the actions being taken 'to meet 
tlris ^oal. 

C onstraints on TRADOC . 

' * TRADOC has been actively pursuing these activities despite 
.shortages of personnel and resources. Another major constraint encoun* ~ 
tered. initially by TRADOC was the varying degree of resistance to change 
in the training environment. Thus, TRADQC is .in the position of riot only 
managing a very large and complex training system, but also is actively ^ 
at tempting to modernize , even revolutionize , Army training in the face of 
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limited resources and resistance to chnnpo. Further^ because there in 
strong sens^ of time urgency in accomplishing these tasks^ the raanagoment 
of TRADOC's planned activities is even more arduous. 

It is within this context that TRADOC has contracted SofTecl), Inc. ' 
to apply its. Structured Analysis arid Peslgn Technique (SADT) to the problems 
of managing rapidly occurring innovations in, training methods. SADT was to 
be used in this study to Identify changes In Army training required to 
significantly Increase combat .effectiveness, describe how Army training, 
testing and evaluation programs would op^er$ite after the proposed changes f, 
are accomplished, and to plan how the 'changes In Army trainirlg would be 
undertaken and how progress would- be monitored and ' evaluated . 

Traditionally, Army training has not been viewed as a large system 

. ' S . . . ' ■ 

problem, but rather as a composite of many smaller problems. These sm£|J.ler 
problem^, usually have been solved by a particular organization within the 
training command. A "good'' solution often o{)timized the particular organ- 
ization's objectives at some expense to the goal of achieving overall 
improvements in Army training. The m^jor hypothesis of this ' study was 
that defiriing military training as a Ikrge system will provide the basis 
for .Tio/e effectively solving, small problems within, the context of oVerall 
military objectives. 

More specifically, TRADOC participated "in this study to improve 
their analysis ani/ planning of the training system. Special emphasis is 

f 

gi/en by TRADOC to the problem of describing the interrelationships between 
the many training Innovations being envisioned and in planning. h,ow those 
changes should be accomplished. Once developed, it is hoped that the model 
of the Army training system can be used to guide -^n<^ control system 
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development from its current status aitd cnpablllty to the 'status* and 'C^pa- 
blllty desired. Aft^r the training system is fully Implemented, the 't^bdel 
could^' potentially be 'used to control an<^<manage the system. Finally 
model produced could become the ''standard" against which to compare current 
system functioning. Faced with the problems cited earlier, and in hopes 
of achieving" the goals described here, TRADOC entered into the joint pro- 
ject with SofTech and ARPA, which is the subject of this evaluation report. 



IV Description of Project Ta^ks and the Eyaluatipn Time Frame 

Prior to Actually applying SADT to Army training and testing 

programs, ''lead time" was allowed for establishing a point-of-contact or 

.J " ' ' ^ 

llason officer within TRADOC to functfion as an Interface between TRADOC 

and SofTech. This person was to be responsible for setting up facilities, 

establishing appointment schedules, and coordinating/sof Tech and TRADOC 

efforts. A major portion of this preparatory activity was devoted to 

training selected TRADOC personnel in reading and authoring SADT diagrams. 

These Individuals were scheduled to play an active role in most project 

tasks, particularly as commenters. 

Proj ect Tasks : 

. The plan for applying SADT in the TRADOC environment consisted 

of five project tasks. A description of these tasks as originally proposed 

appears below. As the project proceeded, it became necessary to alter 

Tasks 4 and 5. These changes will be discussed In a subsequent section 

of this report. . ^ 

Task 1 . In this task, SofTech analysts were to produce a 
model describing, at the overview level, how the Ar^ training program 
would function after^the training innovations envi^^oned by TRADOC were 
actually incorporated into Army practice. The principal purpose of this 
initial effort was to define the ,types of^ changes being planned by TRADOC 
and to de^terraine how these changes, were related to existing training methods 
This initial descript|.on wds to provide a focus for Task 2 work and serve 
as the basis for Task 3. ■ ^ 

Task 2. In this task SofTech analysts and TRADOC officers 
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trained In SADT were to develop .lolntly a description of -a total weapons 
system., The objective of thia task Was the derivation ofVthe training 
requirements necessary for th^t weapon system to be maximally effective 

i 

on the .battlefield. It was belie^d that jinly by looking at a particular 
weapons system as a tfotal sy^tem^ouicl that system's training requirements 
be jfully understood. ' • 

The Army's major weapons systems, such as attack helicopters ^ 
field artillery and tanks, are necessarily conce^l^allzed as dependent 
components within a larger integrated combined arms force. ..Because the ^ 
tank force is a relatively independent system requiring aft increasing 
amount of technical -training for its crew and support elements, it was 

r chosen as the focus of Task 2. The results of a thorough analysis of the 
training support necessary for this Weapon system was expected to be 
representative of what would be found in analyses of other systems. .The ^ 

' insights gained in working throuf^h a.n SADT model of this ""particular 
system wete expected to be ofv> tluf' i n de^f ermining the optinium structure 

of the entire Army t/raining and evaluation program, which was the ^ 

ft * 

c , ' , ' 

^ocus of Task 3, '^ . . 

Task 3 . In Task 3^ SADT and TRADOC personnel were to develop 
Jointly an SADT in^del of the neW Army training and evaluation system as it 
should- -Unction after training innovations envisioned by TRADOC were actually 
incorporated into the Army^'s standard operating procedures. The major . 
input to this effort was to be the general overview model of the Army 

training system developed in^^ask 1, and the SADT model and associated 

■ , » 

reports of the analysis of th^ tank system as a total weapon system ^developed 

.• . ^ • - 

- in Task 2. , . 
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The result of Task 3 was to be a model showing in increasing 
levels of .detail all of the activities that comprise the training system 
of the future. These activitie^s were tc include such tasks as course 
development, budgeting for the Army schools, measuring training effective- 
ness, determining avenues. of information flow and feedback between the ^ 
various organization^ coipponents of the training system, evaluating the 
impact of information and feedback on the training activities that actually 
occur, measuring cost effectiveness, and evaluating the new conceptualization 
of training itself. . V 

It was expected that this model would facilitate a better under- 
standing of ^what the Army was to achieve and reduce the amoupt of time 
spent planning, managing, and accomplishing these achievemrents. Additionally, 
it was believed- thiat the model woulch provide . a format which would facilitate - 
communication to'aiid from the various^ proponent schools and upper echelons 
of the Department of the Army. 

Task 4 . The ot^jective of this tasik was to be the development 
of an innovation plan describing how chan^i^C^n Army training would be 
undertaken and how their progress wpuld be morvUj^red and evaluated. This 
pl^n was to be prepared iri two complementary forms: an SADT planning model 
which shows precisely the interactions between the various elements of the 
plan and an implementation schedule prepared by the Critial Path Method ^ 
(CPM)^. 

^SofTech was to assist TRADOC staff members in developing a 
model of the innovation plan and was to produce the CPM schedule. The 
SofTcch deliverable was to be a report presenting the'm®del, the resulting 
CPM schedule, and a summ^rry of the algorithms used to perfotTn the model 



to CPM converaion. . * 

Task 5, This task was to consist of two distinct parts, the overall 
evaluation of this project (which is reprei^ented by this report) and a plan .. 

J 

for wider Department of Defense (DoD) use of SADT which was to be prep'ared, ^..^ 
"by- SpfTech with the assistance of the project evaluator. ' The overall eval- 

>!■''•■ \ ■ ' • ' 

n whethet changes in l\rmy training 'actually occurred 
•"^ . ' ' * • ■ . 

as a result of the project and whether these changes could i{j[lcrease the combat 
. ■ ' ' ■ • ' ^ ^ . . > ' ■■ ' ■ 

readiness and battlefield effectiveness of the Army. This ei^^kluation was 

to include an SADT model, produced by the evaluajtors, of the overall evalu- 
ation effort. 

If the final evaluation indicated that the project goals were. 

met, SofTech was to prepare a plan for wider ,DoD use of SADT, addressing 

four key issues: ' . 

the types of applications where the DoD could 
i realize the greatest benefits and ah estljnate V 

: of the magnitude of the benefits; 

the types of documentation and courses required 
if a significant number of people Were to learn 
the technique; . 

" ' areas where the methodology should be enhanced 

to improve the quality of the results or its 
, transferability to a wider audience; 

the types of automated tools required for the 
•Qxploitation of the methodology on large projects. 

SofTech expected that the evaluation process would indicate area^ 

in which there were problems : in applying SADT -methodology at TRADOC, areas 

where the technique must be further refined, and areas where alternative methods 

might be pref el^able The plan-^was to indicate how any deficiencies in ' 

the methodology might be corrected prior to wider DoD use. 
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Evaluafclon Time Frame ' ^ , * 

The evaluation of this project was divided Into three phases, 
with the results of'-each phase presented In an evaluation report. These 
phases are: 

Phase 1 > This phase covered Tasks 1 and 2, ^nd 
> cojicentrated on determining the :usef ulness of 

SADT .In modeling Army training. A major portion 
of this effort was the Identification and' development 
• of evaluative dimensions on which the ef f ectlvehess ' 
of saJdt and the usefulness of the models produced 
wre to be assessed . The evaluation report, submitted 
six weeks after the conclusion of Task 2 v^as to 
serve as input In determining whether i or not to 
continue the project as planned. 



• 



Phase 2 . This evaluative phase was concerned prl-i 
marlly with determining the Impact of the Task 3 
model. Because the mbdel produced In Task 3 was to 
represent the bulk of SofTech's work for TRADOC, 
an evaluation of the perceived usefulness and Impact 
of this particular model sefemed appropriate prior 
to beginning the final (Phase 3) evaluation effort. 
Further, an interim evalualzion deliverable at the 
^^onclusion of Task 3 would provide an opportunity 
to reconsider and update the findings and conclu- 
sions of the Phase 1 evaluation. 

Phase 3 . This evaluative phase was to: (1) re- 
examine the results and conclusions pv^ the f irst5r ^ 
two evaluative deliverables, and (2) focus upon'^^ 
the intermediate and long range effects of having''^ 'S^'Sif 
developed the models prodxiced In Tasks 1, 2, 3, 

and 4. Primary emphasis was given in this phase . 
to whether the models had brought about a better 
understanding of what needed to be done to improve 
' Army training, whether TRADOC personnel considered 

' SADT a viable approach to their problems and a 
determination of whether the work produced during 
this projiect would ultimately n^ake a difference in 
Army training and evaluation procedures. This 
paper constitutes ^he final evaluation report of 
this project and summarizes the procedures and results 4 ^ 
of all three evaluation phases^ • 

Figure 4 Indicates the relationship between project tasks . 

and evaluation phases. , 
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Figure ^. Temporal relationship between project . 

tasks and evaluation phases, ' j <|||^ 
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• !; ; f -The following jwsctlons will repott the approach, specific procedures, 
resulcis and conclusions for each of the three evalulldon phases. 
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V, Phase 1 Evaluation Procedures and Results 

•■..-> 

General Approach. ' v. ; * > ' ' i . i 

The proposal! submitted to AKPA by SofTech specj^fied an^ -evaluation 



.e|:£ort which wduld' ;pr(5y,ide ARPA with: , 

(1) an assessment of tbe utility and effectiveness 

of the. Structured Analysis and Desigd Technique- . ^ 
(SADT) in.the training command environment; and • 

' 1 ■ ,'• ' - / . ' 

^ (2)\ an evaljaatic/n of the impact of the i>roject on v . 

\ TRAD0C*8 Ability to identify the types of changes % '"^ . 

required in Army training , and to Vdevelop *'a practical ^ 
. plan for carrying out those chahgfes. 

' The general goals of the evaluation effort (1 and 2 above) and 

the' experience accrued by the evaluators during an orientation visit to ''^'-'y 

TRADOC Headquarters suggested two major areas of inquiry for the Phase' ^ 

evaluation. The first addressed tlhe utility and effectiveness of the . 

, technique Itself and encompassed such questions as: Are .SADT diagrams. 

accur^ite? Does the SADT process promote communication between diVerse ' • 

personnel in the project? Does SADT help achieve clearer unldlerstanding. 

of the problems at hand? ^ » . . . 

The second major ' ai^a of ^nqiliry focused \on the interface b^ween -- 

the technicj.6e and the particular environjnent in which it is applied, in 

this case, TftADOC . The value of the tecririique itsi^f is incorifeequent;ial if; 

the technique is not successfully implemented. Be^rcT^e SADT is depeiideh't 

l/j^^a the,9apperation'ah of personnel in the client crtrganizat ion , 

v.VmQt'glnal implementation in an unrecept;,jLve environment or a Jack of' uniform 

support cou]-d a.ffect^'the utility. and effectiVenes the technique Itself/ 

Thus, two evaluation questions emerged* The first, addressed' the 

./.value of SADT; the second addressed the value of the technique as applied 



■ ih ao environment' like TRADOC. \ThiH second que»tion was dependent on; 
the degree of Impieraentation and the amount of support the technique , 
received* from- the client org Jflzatlqo, TRADOC. Degr^^ /p.f implem^^ 
could^be mdasured by various behuyloral indices of participation in thW 
project' by members of TRADOC .K lhe value question was not so easily 

ahswered. Glveh the avalhlble resourc-ejs an#t frame for the present 

\ . _y % ., :■■ ' . " ' ' • ', • ■' • 

ey^iluat^^^^ ^ef f or^ , the' perceived effectiveness of the technique by partici- 
pating^^ TimboC. of fleers was given primary consideration in determining the 
value ot ^ADT. ' ^ =, 

The remainder of this section fattempts *to answer these quest ions 
by specifying the evaluation methods and procedures followed, reporting' • ^-^'IjiS^ii 
and discussing the resuLts bStain^4v^^^^^^^^^ conclusions based on ■th9;- 

^iEsults, and providing a summary >pf -ctltic;al whlch-a'l^fefcted, the 

implement«^,t;lon of SADT in the TRADOC enyit'cyhm^nt; ^^^^^^^^^^^ "k^^ 

Methods arid Procedures ^ ' , '\ 

/yrr ' A. , Site visits .; :.Xhe evaluation team (Dr. Gary t). Borich and;vV'%4'^^ 

Mr. \Ron Jemelka) visited Softech ' s Wal tham, Massachusetts facility and TRADOCf /" v 

■ ' ■ ■ ^ . . . 

Headquarters at Ft. Monroe, Virginia in August, 1976. These visits provided H' ' 

inital contact with SofTech and TRApOC, ai;id j^roylded a realistic framework / 

from" which to plan ??iibsequGQt 'evaluation effort Additionally, the eval- ^ 

'uatlon staffi was provided training In residing and authoring SADT diagr'ams 

during these sit]e.;.\7isit:s . ^ ' 

B; ' The development of evaluative dimensions . ' The next major 

cValuat4i;?.n^ effott ' was the development of evaluat;ive dlmensiofis to provide a 

fr ;'T\ework for tl>G ass'cf^f^meiit of S^^Bt . The. rationale for developing ' thfe ]evaXuative . 

dlmenslonSv' was taken from the technical proposal submitted to..'ARJ?A.^y SofTech, ^ 
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The approach used in developing these dimensions was to analyze the 
benefits claimed by SofTech to occur at TRADOC as a result of this 
contract. Because the selection of the evaluative dimensions was critical 
to Subsequent evaluation efforts, these dimensions were derived directly 
from t^e contractor's claims, to protect the evaluation from being biased 
by criteria which were either unfair to the contractor or insensitive 
to Army needs and expectations. Deriving dimensions from claims made by 
the contractor in the technical proposal seemed ultimately fair to both 
SofTech and TRADOC,. because SofTech wrote the proposal (made the claims) 
and TRADOC accepted the proposal as documenLat ion of what was to be 
delivered. The only constraint imposed by the evaluator in selecting 
evaluative criteria was thaL each dimension chosen had to be measurable, 
given the resources and. lim^ frame* allotted to Lhe evaluation efforL. 



process of deriving evaluative dimensions began by clustering similar 

t lalm^; tpgeLhor. Aflor LhLs flrsL f I us Li»,r i ng , each clusLor was scrutlnlv.ed 

fMrt'lior Lo de tcM 111 Lno if (a) thiM c were slgui f I caiU: over lap between clusLcrs, 



r<ulirally dlfforent cLiinui in any one cLuster. En some cases, cliiHtors. were 
I'onibiiKMl and, in oilier cast^M, t'Uistt*rM wtM*e broken into vvAiilcd subc lusters. 
'I'he overriding; c^r Iter Ion for the compos iLlcm oi' c.lusLors was that each be 



KacU rlusttM- was t hcMi ;inaly:'.<*d aiul a ^;e^^t^f*^c. title chosen to cover 
«i 1 1 I'laiiirs in Lliat clujt.M', In this mannt'r, each cluster jevolved into an 
(w.iluat ivc d I iiUMir; i on . rhes(^ cvaluativi^ diiiKnisicnis appear In Appt»nd i x A. 



Jhe technical proposal was analyzed and each claim was noted. The 



(h) t lie re were only a few claims In any one clusLer, and (c) there were 



u; independent of ihr otlu'rs a^^ pnssiblt^. 




C. Data gathering strategies . Given the /jtime and resource limita- 
tions of the evaluation effort, the most direct manner ^n which to evaluate 
SADT in Tasks 1 and 2 was via the perceived effectiveness of the technique 
by TRADOC personnel involved in the project. The^ most straightforward 
strategy would be to ask these individuals about various aspect;s of the 
technique relevant to the evaluative dimensions and what benefits they 
believe have accrued or could accrue to the Army as a result of its partici- 
pat ion in the projec't. Therefore, (lUestionnaire was developed and admin- 
istered to TRADOC personnel and then followed with a structured interview 
focusing on specific re3ponses to the questionnaire. The interview also 

provided the opportunity to pose general questions about the value of 

> ,.. • ' • ^) 

the technique and the use of the- technique in the TRADOC ^vlronment. A 

^o^SSi^the questionnaire appears in Appendix B. . ; 

According to SofTech's deijcriptive literature, one of the necessary 

"subsystems" of an SADT, project Ls a file system and library service which 

(> r^Vini^-t^'' and ctnit ral i /.I's rocord-lceeping and siippc^rt functions and which ' 

y 

allows the complete prt) j oc t *^his tory to be .reviewed. This clerical function 
ensures that documentation of the development of an SADT model is maintained 
concurrent wit h that: dcvelopmcat . A copy of each diagram produced' is 
rcLci Lned on I i Lc, and a I L Interact Ions between Individuals in the project 
are documeuttHl. 

This project tilt' makes it, ptisslble to delefmlnt* tlie iriibject area 

of a kit. (a .^;ei:i(.\^; ot~ related SADT di,a^;rams), number of'dlagrami^ in 
the kit, who t:ho kit was sent to for conmients, hc^w many Limes kits were 
coiimuMUfd 1)11 ami rctiimed, and how ot lcu I hi' authot^ rt'turnetl the kit to 



the re\(d>X. From these data, one can determine whethe'r the activities 
described in the proposal actually occurred. While - designed to enhance 
the review of project decisions and development, these data were important 
in determining if the technique had been successfully implemented at ' 
TRADOC headquarters. 

The project file at Ft. Monroe was examined to determine if this 
library function had been maintained as claimed. The number of diagrams 
produced and general content of the diagrams were noted. These data » 
then were used to prepare- frequency tables for all relevant process bejiaviors. 
TRADOC participants were queried at random" to determine the degree of 
agreement between perceived level of participation and degree of participation 
documented iii the project file. ' a 

D. Summary of procedures . Tlie Phase 1 evaluation procedures, 
yielded three types of data: 

(1) proces.s data - obtained from examination of the project file; 

(2) cpia^t i tat i ve data - obtaLned f rom the quantifiable question- -r. 
n;i t ro responsivs (ryC.v Append Lx T loms, I through 38) ; 

^ CO (]ualiLaLlve data - obtaLued from responses to open-ended 

(juos t lonnairc Items (Items 39 tlirough 44) and from responses 
to iiiLervlew probes. ^ 

. At LlM' the resul ts of eacli type of data were prepared they were 
cdnibuiLul where appropriate to annwor tlie two basic questions v|liicli were 
Llu> fix'us n\ the PhasL^ I evaluation effort. Tlio ILrst (piestLon concerned 
the v>iluo ov merit of SADT Ilsclf. The second question addressed the 
i lup'l tMUcu tat. iiMi n\. SAIVl, a r L i c u i a r 1 y I n an tnjiv Ironment such as TRADOC. 



The data frorii all three sources were interrelated prior to drawing concltir- 
8 ions about the value of SADT. T|je interaction of these three groups 
of data illuminated both the technique itself and ^he effect of variables 
which moderate the effectiveness of the technique. 

Results ^ ' 

• ' ■ , ■ 

This section presents results of the Phase 1 evaluation. Data 

on the implementation of SADT during this phase will bo presented first 

followed by. results from the questionnaire and structured interviews. 

A. Process dcit£. The first step in this 'phase of the evaluation 

was to determine whether the roles of the TRADOC and SofTech personnel 

■ # - ■ 

specified in the contract proposal had actually ^een performed. Examination 
of what had occurred up to this point in the project revealed that the 
pro(;esscs and role functions specified in the proposal did occur with 
some exceptions. One notable ommission, however, was that no Senior Review 
CoirmLttcc was reported In the Task 1 report. SofTech explained the absence 
of thki Senior Review Cominittee by noting: 

As fewer changes ovrere indicated by comments 
received, the diagrams would normally have been 
sent to a fielectod group of comment ers referred 
to as the Senior Review Committee., This committee 
winild be rcquestLul to review bhe.^e diagrams and » 
approve them.- Sineo the comment cr.*5 included the 
itiembtM-s of the review committee, ^'^nd sin^e this 
' • ' - model will be revised in Task 3, the formal senior 

review was (miittod, (SoF'i'ech, XncJ , 'l'a«k 1 Report , ^-5) 

< . / 

A secontl d L^c rop^i^^'^'y InUween contrac/or ^-ntonts and cond i tions 

ol).sorved by t ovaluat:orH w<is; tliat some individuals performed mnl-ti-ple 
roles. (^xamj) 1 the t:,ochni('al proposal j^pe.cifLed tliat the individual 

assij;iUH[ as iho TRADOC iatorra((> was to ho rospons i,h l.e for time ccnnm i tments ^ 

■ ■ ■ ; ■ 



selection of ataf f for t?raining and participation, and for ensuring comple- 
tion of TRADOC deliverables. Al|p;^ugh these functions were performed 
initially the role was modified early on in the project and ultimately 
became nonexistent. This individual was given other assignments unrelated 
to his responsibilities' as the TRADOC interface and was asked to perform 
other duties vpthin the project that limited the amount of time that ^ could 
be devoted to the interface task.'^ * 

Another point concerns the sources of inforfnation for SADT 
authors. Although tjie number of experts interviewed was at least ten (as 
specified in the proposal), discussion with SADT aujjTiors indicated th^^t 
early Ln the modeling process, significant information was gained by 
reading technical and progress reports on training and the proceedings 
of several conferences on training sponsrred by TRADOC. Thus, the SADT 



process was. not implemented initially; but rather following a schedule' 

of background reading which undoubtedly made the. subsequent author-TRADOC 
interviews :^pre meaningful. This secMus to* be a dt^sir^^ble and natural 

activity prior to tot+elmg but -no .mention is made of it in the SADT authoring 

procedures. 

, . KxamLnatibii of the ovenLs surrounding Task 2 of this project 

rovoaUul ^;everal d Lsc ropnut' ios from those spccifiod in the teciinical 
proposal. The most notab le tuvo I vcs the full time assignment to Task 2 
of tiwo TRADOC personnel aj; SADT authors. The SADT training provided these 
individuals was not timely, or necessarily complete reiati.ve to the Task 2 



'^Another iudivLthial with scvcmmI. functions was tlu' on-site project 
manager for SofTocli. fn addition to management responsibilities, he 
was one of the two autliors provLded by SofTerli for Task I.' He also 

assumiHl some oi the fnnctions of the TRADOC Interface when necessary. 

VI 



J • ' , . " r 

effort. Only one TRADOC author received formal aiitli^ ti^alnlng, this oc- 
curring during''%e second half of 'task 2. Although the second individual , 

^* received some on-the-job trailing to prepare him to author diagrams, this 

■f • • '■ ■ ■' ■ ■ . 

training was reported by 'the two ScJfTech authors on site to be minimal. 

Other than those nojf^ed above, the prescribed roles listed for 
Tasks 1 and 2 in the technical proposal were carried out as specified. 

Examination of SofTech's descriptive literature reveals that 
several ^related, classes of behavior constitute ' trhe SAI>T process or disci- 
pline. Two of these are tjie iterative author-comraenter cycle, which is 
the vehicle for communication between SADT authors and commenters in the 
client organization, and xnainteaarice of the SADT project library, which 
provides complete documentation pf the project history. 

The claims made by SofTech imply benefits such as enhancec| communi- 
cation, understanding, and involvement by personnel in the client o^gani- 

c 

zation, and casts SADT as a he.uristic device which will aid in the solution 
of problems. The most central element of the SADT process Is commenting 
on a kit. Because commenting on diagrams Is the only official contact 
between SofTech personnel and , individuals in the client organization, 

'this activity must logically occuv before any of the above-mentioned 
bouytlts accrue. . 

Table 2 presents datca on y^li issuance and return by rank of 
rommenter for Tasks 1 and 2, Al L the data presented were obtained from 
the Task 1 and Task 2 reader kit eovershcets in the^ project library file... 

^Column LL represents , the numbe.r ot kits each, reader was expected to 
c:ommei\t on. Colmmi 1! t iidlcatcs the number of times the reader actually 
fommeiited on and re turned ^hc kit to the author by tlie t Lmo prescribed. 



Column IV represertts those instances in which a kit was returned but late. 
Column V infilcates the number of times a kit was sent to n rendei* for 
comments but never returned to the^author. Column VI indicates the number 
of times it was unclear a^ to what actually transpirec) with any one kit, 
i.e., it was not possible^ to discern whether the kit had been coiranented 
on or not. This is clearly a failure of the librarian function to document 
precisely what had transpired. * 

■ :Totals at the bottom of Table 2 indicate that 104 or 54% of the 
kits issued to readers for comments were not returned to the author. 
This level of participation by TRADOC personnel in the iterative author- 
commenteb cycle of SADT was considerably l,ess than SofTech had originally 
anticipated. . N 

The large number of kits not returned led the evaluators to ask 
a SofTech project member about these discrepancies. It was indicated 
that in the later stages of the modeling, the interaction between SAUT auti^ors 
and the Senior Review ^Commit too was often' verbal and final approval of 
clva^'.rams was obtained in a conference setting. According to SofTech 
desc r i p t ion^; , however, a conference between commenter and author is - 
reserved for the case in which dl f fcrences cannot be resolved on paper 
and all intorac t Loniii , for wliatcvcr iSurpOse, arc to be recorded. The 
procedure used appears to be at variance with SofTech' s des.criptive 
'literaturo. . , 

In attempting to explain this discrepancy from standard procedure, 
a SofTL*ch project member explained that working in this environment was 
at tlmns too fiectlc i'ov him to perdjcSrni c ler ica I duties. Another SADT 
author stattul that to his kaowlcdj^o, hv logged Mil rcciulred data on kits 
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Table 2 



Commenters by Selected Process Behaviors for Tasks- 1 and 2 
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TRADOC 


Number 


Number 
of Kits — 


Nuinber 
of Kits 


Number 
df Kits 


Number 


Number 

o r 


Commenters 


of Kits 
Received 


Response 
Requested 


Returned 
to Author 


Returned 
Late 


not ♦ 

Returned 


Incon- 
sistencies* 








on Time* 








Generals 


: 6 


.4 


2 


, 0 




1 


Colonels 


93 


■ 76^ 


12 


5 


57 


2: 


Lt, Colonels 


30 


27 


11 


4 


11 


1 


Majors 


18 


18 


10 




,.7 


\ ' 


Captains 


59 ■ 


V 49 




9 


24 




Civilians 


19 




5 


7 


.■ - .V 4 


■ . . - 


:RAD0C Totals ' 


. 225 


191 


55-29% 


25-13% 


104-54% 


7-4% 



^This column represents the number of times it was unclear from the coversheet information whether 
the kit was returned to -the author by the conuuenter and appears to be the result of poor book- 
keeping. The total of- columns 3, 4, 5 . and 6 should equal column 2, ^ . 
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he authored and issued, but added thatk'perf orming these functions det;racted 
froiin his authoring activities. Vfh^n^sked specifically about two indir- 
viduals.for whom records Indicated they had returned less than 10% of the 
kit3 issued to them', the evaluators ^received verification that the figure 
seemed valid. Despite the tendency of SofTech authors not to record 
final stage^^^^pprovals of tKeir diagrams^ it seemS ^reasonable to accept 
the reader kit cover -sheets as estimate^ of the interac tioi\s that went on 
during the project. f 

■ • • ■ - . ■ , . / 

Another area wh^re SofTech' soi'iginal expectations of TRADOC 

. ■ ■ ■ , t •■ 

pcjrt icipation was not met was in the assignment of two TRADOC personnel 
who were to become authors for Task 2. This assignment seemed critical 
to TRA^ipc's -"Internalizing" the SADT procedures. The fact that the indi-..- 
viduals were not provided in a way that they couJLd both be fully trained 
and assigned to the SofTech project full-time as expected raises two issues • 

The first pertains to the productivity of ■ project personnel' 
during Task 2. Given th/it one. of the individuals assigned was never 
formally trained and the other was trained threes months after the Task 2 
efft^rt was begun, it would be unreasonable to- expect the same results 
and degree of internalization of the technique had initial expectations 
been mot. To , quote the program manager of this project, "So f Tech certainly 
would have been mur.h more guarded in predicting the impact of ,the work 
at 'I'RAnOC ir the aftuaL level of Army author participation could havp 
hovMi pr-ecMV:tetI at iho t ime the proposal Was written." • 

The MCUcMul issue arising; from TRADOC ' s failure to provide the 
min,^'n.'»M- originally a^^rcu?ci upon eoncerus the evalnat: ion of the trausler 
of ' ' . i s' Lechui) logy t ri>ni SofTech fc) rRADOC, Oetermining the extent to 
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wliich Army personnel can be. traine^^'.in SADT methodology seems critical 
to 'ietermining the applicability of the'^echniquef to other environments 
within the Department of Defense. Unfortunately, only one author was 
trained. Wh^.le this indivldua] was^bnsid^rea a good SADT author by SofTech 
oerr jnnel^ he was viewed by his superiors at TRADOC as a unique individual 
with considerable talent and potential, especially in the area of 
operations research. One of f icei; ySt^ted during an interview that th*is ' 
individual ^'is not representative of the population at TRADOC." Certainly 
the^e are competent individuals who can learn th^e technique., but theiS: . * 
productivity in using SADT to model program functions and the range of 
individuals capable of learning the technique in DoD environments .cannot 
be determined from the present data. 

In summary, the process dc^ta collected indic'ates that the SADT 
nrocess„was not always implemented as specified at, TRADOC through Task • ^/ ^ 
Participation by TRADOC personnel in the iterative author-commenter cyclQ ' • 
was apnroximnLely half that expected. This reduced availability of TRADOC 



personnel for tiraininp and authoring of SADT diagr^amfe limited the potential 
to institutionalize tlic teclinique at TRADOC. Thus, final and definitive 
conclusions about the success of transfering this technology to a DoD 
environment; cannot he made from these data. 

R. Qu antitative data . This section reports responses to 
multiple choice and- Likert type questionnaire items givc^n to T.llADOC ; 
or icers working wltli the project. The. questionnaire was completed , : 
by 13 of 16 Task i and 2 commenters. Statistical calculations indi- 
cate that the (juest i onna f re reliably measured some chhtacXerist Ic of 
the respondents, and that the IndivWuai Items were producing slmiiat: 



patterns of responding in dTjYerent individuals ( ot = .95).* The same 

was true for eval-uative dimensions II. ( a =« .86), IPI^ ( a = .96) and 

Jllg ( a . 96) and to a lesser extent for . dimension 11^ ( a = .71). 

The pattern of responses « for^imensions I , I , and II were mare heter- 
* . , Ad B ' ' 

ogeneous and of questionable reliablity ( a - = .35, .45, and i51 respect- 
ively) . » 

. T^bie 3: present^'/the number pf^^respondents, the mean (X) and 
the standard deviation (SD) of each quantitative item on the questionnaire 
The major intent of each item Is included and items are clustered by eval- 
uative dimensions for easy reference. * > <. 

Two . conclusi9ns can be made from examination of questionrtaire data. 
The first concerns the value of SADT as an "engineering drawing system 
for systems description. " ' This view of SADT ,y presented by SofTech in* the 
technltal prbppsal, stresses standardized graphics, controlled document 
revision through the provision of procedural ^mechanisms for audit purposes 
and effective communication between the originator and> user of descriptive 
models . - ' , ^' . 

The results of Subscales I,, 11,^, and II„, indicate^ that as ' an ' 
/ ' A A D 

cngl.neoring drawing syj>tcm» SADT is' viewed positively by TRADOC personnel, 
riic general consensus is that the SADT discipline, which specifies the 
format t)f a^l commun i tvU. Lous » . controls the routing of diagrams and 
ostabl LsliOvS the t Lnio Crame Ln which these behaviors occur » generally 
results In accurate, highly readable diagrams that communicate the 
suF)r, tant i ve components ot complex problems. 



" *Alpha ( a ) reliabilities calculated wit:h the standard- Kuder- 
Richardson fbrmula (see Guilford, ,r.P. Fundamental statistics psy- 
chology . and educa t Ion . New York : McGraw-IHll, \965, 458-460.) 
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I tem Means and Standard Devial Ions for TllADOC Headc|u;WVerf^ v Comment ers 



Items 



N • 



Mean . 



Standard' 
Deviation 



I. .COMMDNICi^TlVE QUAj;i LTY 



Ai ' ^of- diagrams . , 

1. Respondent adequately trained, to ' , 
read dicigrams? 

2, Respondent adequate-ly trained to 
comment on diagrams? 

3a*. First drafts of d:Lagr7ims clear aiid 
unambiguous? 

■3b^. Revised versions of diagrams clear 
and unambip»uous ? 

3c. Final versions, of diagrams clear 
, 7 and unambiguous? 



13 



13 



12 



13 



12 



4.08 



3.92 



3.75 



4.31 



0.95 



0.86 



0.87 



0.48 ^ 



0.51 



10. 



Wcuild written text explaining 

diagrams be, help f ul ? : V , 12 

Should discussion be a part of the 

SADT process? '^^^ - ■ 13 

A re cl i a g r am s an e- f f e c 1 1 ve .way tV^-*^ 

conmiun i a t i ng 1 3 

* / 

r ro j ec- 1 bone fits /'i I L r Lb iiKib I c» L o- 
fliagrams or to ^Vrocei^^ses i^imu lifted?"^ 



Moan and standard d-evlation f(^r 
Subs(\i Lo [ 




A 



ot Pr(uu»ss 



3.00 



4. 38 



"4.23 



3.44 



3.9 7 



1.21 



0. 75 



0.60 



0.53 



0. 75 



7. Have diagrams gLMiorattnl couuinuiL"- 
\^ • ca t: i nn ■ aiuonf, indi vidua Is Lnvolvod 

( i . t' .^^,>/?tskt»(l to bo oommontors) in 

pro J ec t ? i 
8: ' llavi,» dL.ij\i\ains ^;t»ne raL ed coniiiumi- 

cat ion a'iiu.n)y,. . i nd Lv LtUia Is np ( 

inv()lv(»(i (X.t^., lu^t asked to bo ■ 

( u );uine n 1 1» r s ) in p r o j or t ? 
9. Have (Hag, raiii.s hi»lped tOt-iis at ten- 

t icni (Ml v.i r i al> 1 es ai> t rons I dt» rtul 

p re V i i ) lis I y7 " 



13 



3. 38 



1. 85 



L 38 



1.45 



0.99 



bO. r ro j e(^ t . beutM' i t s .it- 1 r i but al) 1 e t o 
tl I ag,rani5'» ox to piocer.ju-s 
:it I mil I. It Oil?* 



44 ' p. > 



3.44 



0. 5 3 



Table ,3 (con t i nuod) 



Items 



'30. Were others consulted'^be f ore com- 
menting on a tl Ingram? |f 



II. 



12 



Mean and standard deviat ion for 
Subscale 1^ 

QUALITY ' 



6. Are diagrams an offt^^giiivG way of* J 
communicating?* *" . ; 

11. Have diagrams accurately rep resented 
content modeled in Task 1? 

12. Have diag^rams accurately rep rcsen ted 
coyitent mcnleled in ,Task 2? 

14. Confident that autho r-comiiien ter 

cycle guarantees accuracy and • y 
cbmpie t^ficss ? 



13 



11 



13; 



f^tcan and stanclai'd dovi.ation For 

Subscale U » — ^ 
A 

13. Does aut- lior- coinmevi ter cyc l e ensure 
quality of diag.raiiuj? 

lA. ( I di'U t t ii:it ;iut hor-comiiuMi t or 

' ..cycvLe j;ua ran tct^ > . accuracy and ^ 
ooinp I e tLuiess ? 

1 

13. < A Ki U 1 d H, I i 1 a ,t c o 1 1 ini u a t s w c, re ' take n 
* L nto acciuiii t ? 



1 3 



Mean 



Standard 
Deviation 



1.00 



i . 01 



A. 2 3 



4.09. 



4.00 



4.08 



4 .10 



4.31 



0.00 



.0. 85 



0.60 
0.54 
0. 87 

0.76 

'0:6 7' 



0. 73 



4.08 0. 76 



13 



4. 77 



0.44 



18. 



(ion I i (leu I in roimueuts made on 
d i ag/r ams ? ■ ' . ' * 

Rcv^piMulout lacktul raifficLtMit cKpt^-»- , 
riiMUH' In inako iiuNiningful couuiumi is ? ^^'^ 



MiMii and .^itaiulat'd dtwiatiiMi lor 



Suhsi • .a 1 r I I 
C. . 1, M i ri t'iir\' 
M). Has (*oiii(*iiuu I iM I iuy^ via authnr- 



I 3 



1 \ 



4.00 



3.42 



^. j; 



0.38 



38 



0.8^ 



1.03 



1/. lias S^XDT, s.ivtul time rc^latlvo to 

other avallahlo apju'iKudu'-s to t lu* 
p roh I cm ! .y^*^ 



3. 36 



1 . SO 



Table 3 (continued) 



:-e'ffiL 



18. Is SADT cost-effifcient "relative to 

other techniques having some purpose? 

J.9.^^^iPercentage" of time sayed tis'inp SADT ^ 
^ : ''^on design and analysis task?t 

37. Has the progress m£ide been worth the 
time spent?'^' ' 

Mean and starid|ii?ti deviai^ion Cor 
Subscalo T ■ 



111 . USEFULNESS- 



N, Mean 





12 



3.67 



0.26 



3. 50 



3.56 



V 



Standard 
Deviation 



1.21 . 



0.24 



1. 38 



.1.28 



"(\. In Tasks L and 



9. Have (U nj^, rams htMped l-ocus attoution 
on Variables not considered 
previously?* . "13 

20. "Bid Task. 1 model provide clearer 

under.^ tan ding of Army training,? ^ 11 

2 1. ' Will models produced thus far^ 

actually he. us,ed in planning changes?' 

22. Will models produced . thus far 

'actually hdp to do r Lve ' training 
. fequlrvMuents? 12 



24a. WU L SADT .incro<u;c 
capability?^ 



RADOC's an 



a lysis ^ 



10 



3:3*8 



3.09' 



2.9:!^ 



3,17 



3.60 



1.26 



0.83 



1. 11 



1.51 



24b. Will SADT increase TRADOC s plannin^^ 
. capab i L i ty 



1.0 



3,40 



1.^6 



•24c. Will SADT. increase TRADOC s manaj^A'- 
mcut capalH l.i.t-.y 



10 



3 . 00 



1.05 



25.. Can SADT help i <1(mi ti f y . t ra In 1 ng 
riV[u i riMiieu t .s ari\u'tlnj; combat 
c t ice t L vencss?^'^ 



3, 7 7 



1 .09 



"27. Ks U practii-al tor-1'RADOC to use 
SADT for de^iiy»,a aiuT analysis 
p vo b I ems , ^ 

' \2 . Wll) .SADT modehi p' rcw i de ** i mpe t us tor 
i-hanr.es in Annv t ra i ni u^.V '^' . . 

\\. Will SADr bi^^uselul In I dent I t y inj; 
i^x.lsl in>> oi");an i /.at lonal 
Inel t icienclt^s?^'^ ' 



12 



3.62 



3.17 



3. 7S 



1. 12 



0.94 



0.,6: 



4(> 



Table 3 (continued) 



Items 



Standard 
Mean Dcv lat ion 



34. Will SADT lead t:o .conceptual ins ights 
about conducting, Army training?* 

36. Would It be uHeful to train some Army 
personnel to be SADT auth{)rH?* 

37. Has the progress made been worth the 
time spent?* 

38. Did Task 2 model provide clearer 
understanding of Army training? 



Mean and Htandartl tlcviatlon for 
Subscale 111. 



A 



B. 



23. Will SAUT' be ef ft'ctively used to 
elaborate Task. 1 model in TasH^. 3? 

24a. Will SADT increase TRADOC* s analysis 
capabili ty ?* 

24b. WILL SADT increase TRADOC/s planning 
o/ipability?* 

. i 

24c. Will. SADT increase TRADOCVs manage- 
ment capability?* i 

25. Can SADT help Identify training 
requi remen t5? affecting combat 
ef fectivene^i^fi.?* 



13 



13 



12 



11 



13 



10 



10 



10 



13 



3.46 



3.69 



3.50 



3.64 



41 



4. 31 



3.60 



3.40 



3.0D 



3. 77 



0.88 



1.49 



1. 38 



1.03 



1.10 

0.63 
1.51 
1.26 
1.05 

1.0.9 



,2 7. Is it practical for TRADOC to use 
SADT for desd'gn aikl analysis 
probl(^m5T?* . 



32. 



33. 



34 



35: 



36, 



Will SADT m.q4j-i^ provide impetus for 
c h an ge^s in . • 'A im)f^ /training?* 

WilU^S/l^frHe fiseful in idsfl^ifving 
e X i s t i n)| , o^gan i z a t i o n a 1 
Lne ff ic||^il^f»^''■** 



I 9 -k 



Will SAD%Iead to conceptual insights 
ab It convicting Army training?* 



Will use of SADT ultimately affect 
training concepts lield by TRADOC? 

Would it be useful to t rain some Army 
personnel to be SADT authors?* 



'Mean and standard do v Lat ion for 
Subscale Itl^^ 



13 



12 



12 



13 



13 



13 



3.62 



3.17 



3. 75 



3.46 



3.15 



3.69 



3.54 



1.12 



0.94 



0.62 



0.80 



1.49 



.04 



47 



Mean 



SPECIAL SCALES 

Value of SADT Model J}B ii_ Reference 

26, Would an SADT moclel of Army Iralninj^ 
be of value as a reference document? 



Mean and standard deviation for this 
subscale 

I n f o rma 1 1 o n O r a sp 

2 8. Confident li\ comments made on 
diagrams?*^' 

29. Respondent lacked sufficient expe- 
rience to make meaningful pomments?** 

e 

30. Were others constiJ.ted before com- 
\ ifienting on a diagram? ff 



Mean and standard deviation for this 
kubscale 

31. Presently, how critical is It to 

implement'* changes in Army training? 



Mean and st a ii cTar cT cTe v 1 a t L on for this 
subscale 



13 



13 



12 



12 



13 



3. 85 



3.85 



^.00 

3.^2 

1.00 
3.71 



A. 62 



^.62 



1. U 



1.14 



0.'58 



1. 38 



0.00 



0.62 



0.65 



0.65 



Quest loan ai re Total 



3.58 



0.93 



Item also represented by another evaluative dimension. 



•k-k 



S,coring reversed for this item. 



•k-k-k 



13 



This Item had only four alternatives, scored in the following manner: 1=4 
2=5,3=2,4=1. ^ * • 

^This item was not in multiple choice format. The mean and S.D. for this 
item were .not included in* th'e calculation of the per item mean and S.D. for 
Subscale Hq* 

This item covers a communicat ion aspect d iscouraged by the methodology. Xi\^ 
mean and S.D. for this Item were not included in the calculation of the per 
item mean and S.D. for this subscale. 
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I stores were lower for the evaluative dimensions that inquired 

into SADT'\s impact or usefulness at TRADOC (I^ , III,, III ). When 

queried aboW SADT d^lagraras and the SADT process £er se , responses 

were^quitef p^Witive (The item means for evaluative dimensions I II" 
■ A ' A' " A' 

and rlgVwei:^3\97, 4.10. and 4.12 respectively on a 1 to 5 point scale). 
Vnien question^ Wdressed the application of SADT within the TRADOC context, 
resulting' 8cores\ wqre noticeably lower (2 . 61 , - 3 . 41 , and 3 . 54 ' f or evalu- 
atlve dimensions- 1^, III^, and III^, respectively). This difference. 
beb*/een tlie context-free and context specif iq views of- SADT .suggests 
that there are characteristic^ of either the technique, the environment-, 
or both, which have limited the applicability and usefulness of SADT In 
the TRADOC context . Data from other sources wilT be employed in other 
^ parts of this report to further illustrate the distinction between SADT's 
generic qualities as exhibited by the diagrams themselves and the 
technique's capacity to become Integrated in the TRADOC environment. 

C> Qualitative Data . Discussion questions from the questionnairo 
and a follow-up interview with respondents shortly after the administration 
of the questionnaire provided the raw data for the following interpretive 
comments. Despite differences in the format in which these two types* of 
data were collected, considerable consis^tency among verbal and wtitten 
comments emerged. The d^cussion of interpretive data begins with a 
presentation of the most general* conclusions that emerged from analysis 
of the subjective comments of TRADOC members. 

The most general conclusion shared by all but a few respondents 
at the end of Task 2 was that It was too early to say whether the 

' ' <> ■• 
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technique had any practical value for TRADOC. While almost all respondents 
Indicated that they thought the technique was useful at a general context- 
free level, they were unsure about Its applicability 'and usefulness 
within TRADOC. In response to the first discussion item on the question- 
naire (Could you identify any specific benefits . which may have accrued 
to the Array as a result of Its participation in this project thus far?), 
the following conunetit was representative: , 

Not yet. Th«Ls has been some concern to me 

since the project started. The potential- ' 

for using SADT as a tool to improve Army 

training is real. Perhaps I am not involved 

enough. ' 

And, in a follow-up interview, this comment was expressed: 

We went into this project to get someone to 
provide a logical layout of the training 
system so we could get a handle on it. Dumb 
grunts like myself haven't gotten that out of . 
it yet. I don't know if SofTech's work will 
be of any use at this time. . . . I ' 11 wait before 
saying yes or no. . ^ 

All commenters could readily enumerate strengths of SADT, especially 

when asked about the technique in a general, context-free sense. While 'V 
o • ■■ . ■ * 

many strengths were mentioned, the following occurred with some consistency: 

(L) SADT requires a graphic presentation of the 

problem which simplifies the problem and 

communicates it succinctly, i . e , a picture 

is worth a thousand words; r * 

"* (2) Commenting forces an individuals to think' 

^ critically about a problem before he can . , 

disagree with a particular diagram; 
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(3) SADT promotes af consensus of opinion about 
|j. .a previously dll-defined concept; ' 

(4) SADT saves time over verbal communications 
because it's hard to "get off the subject" 
in commenting on an SADT diagram; 

(5) SADT illuminates the real causes of problems • , 

Generally speaking, commenters provided positive accounts of SADT; all but 
one individual '•expressed that they believed the technique was a good oiV6. 

When asked about weaknesses of the technique, most commenters 
menitioned aspects of ythe application of SADT and. not generic qualities of 
the diagrams thems^ves. The most consistently mentioned weaknesses were 
that the SADT process was too time-consuming and that commenting on a ki^^t 
occasionalJ.y interf erred with an individual's ongoing work. It can be 
seen that these comments mildly contradict those made about the strengths 
of the technique (for example, #4 above V. This paradox was evident in 
m^ ny *'nf the comments made and indicated that TRADOC personnel considered 
the technique during this r ^; of the evaluation as a good one genejrally, 
but were unsure of the value of its application within the TRADOC 
context. . • 

Several consistent themes and comments emerged from examination 
of the interpretive data which help illuminate the ambiguity on the part 
of TRADOC personnel abi^ut the value and usefulness' of SADTf as applied to 
the complex problems of providing Army training. One of the most consistent 
themes was that insufficient resources"' had been expended on the project 
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thua^'far. Itetn|A3 of the questionnaire inquired about obstacles which may 
have hindered' Sof Tech's productivity ati TRADOC. The following response 
to that question was typicM.: ' - ' \, ' 

The normal worklo«'id at 9CST (Office of the Deputy 
/; ' ' Chief of Staff for Training) is so heavy that ' . 

numerous key personnel were not able to get in- 
volved at the level desired. This resulted in 
reduced visibility of the project.* As the project 
grows' to Task 3, there should be greater involve- 
ment, particularly after the Commander Conferences 
(in December) . TDY (temporary duty involving travel 
away from usual assignment) and inaccessibility 6f 
key TRADOC staff ~ this may have hindered them in 
getting the job done. 

Several other individuals mentioned that tliey had not devoted 
as much time to the project as they should have. ^^i^^^r^may~he one reason 

. ■ . V?r 7 

why individuals valued the technique generally but jwete unsure of its 
use and applicability. 

Another consistent theme in the interpratlve dat,a concerned the . 
command emphasis for the project. Several commenters were critical of 
the army's support of SofTech's work. Much of this criticism was aimed 
at, "'upper level Army management" as ^evidenced by the following comments 
froTn interviewees (all below the rank of Colonel) : ^ 



of cofmnar 



There has been a lack of comnand emphasis. llve^W^S^^^^j^*: 
no pressure £rom my boss. It (participation in T - 
SADT project) has had no bearing on my job as far as^ . 
he is concerned . ' 

There has been littis enthusiasm for this 'project 
(SADT) by upper level management. , 

Whether it will be used here is strictly dependent on 
management .... If managers use it everyone will lise it; 
emphasis has got to come from the top. If this happens 
it will be successful here. 
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Anecdotal comments written in the margins of mulitpl^-choice 
questions indicated that usage and value were "entirely dependent upon . 
command inf l,^u:ence,** and that the amount of time devoted to this project 
"had been insufficient." .One individual indicated that all questions 
pertaining to ^^Ssage were dependent on management and he doiibted that they^ 
would give this project a high , plriori'ty. To summarize, there was. some 
skepticism on the part of lower-ranking officers that'^'upper level Army 

management was actually concerned about the SADT project. 

■ ' . ■ • 

A' third factor contributing to skepticism about the value. and 
usage of SADT at TRADOC was ambiguity and uncertainty shared by most 
commenters of what the project results should be. This impression became 
evident during inital contact and interviews wit^i the TRADOC organization. 
This confusion still existed at the conclusion of Task 2 as indicated 
by ths following comment from a TRADOC colonel: 

...... . ' ■ . \ . 

We don't know. what out* (training) system is. * 

We don't know how we will use it (the SADT ^. 

model of . training) . I'm unclear "in my own 

mind about how I'm going to use it .... There ' s 

no clear perception of how it will be used 

by me or anyone I know. 

A fourth consistency evidenced in the respondent's comments 

concerned the timeliness of the SADT ^proj ect . The project closely paralleled 

,22 

the Total Tank System Study (T S ) whose goals were much the same as those 

■ 'T 

of Task 1 and Task 2. Several respondents pointed out that 3ADT was im- 

• V ■ ^- ^ 

\ . ' 2 2 - ^22 ' 

plemented after the T S study was well under way. Because T S and the > 



SADT project overlapped, . both in terms of -time, courfle^' and general goals, 

the relative contributions of each to what was ultimately learned about 

tank training is' unclear^: ' • " ' ' 

The chief of ficer at the T S group had tfie most" to say .on 

2 2 

t||is issue. He stated that the T S results contained the "meat" of what 
learned about training, but^that the work done using SADT gave clearer 

definition of those results. He felt ,that if SADT had been introduced prior 

' '22 • * 

to the beginning of the T S. study, it would have contributed to the results 

addressed the timing issue in the following manner: 

. <^ 

: * I was immersed in. real problems; we didn't 
have time to learn new techniques. ,It was 
initially a hindrance time^wise and procedur- 
ally .. .After the bulk of T S was finished 
we had time for Sol^Cech and the -value of the 
technique became obvious, 2"^5 communicates 
the "big picture" of the ?f* S results. 

: Because the SAdt process was instituted , when it was, and giveA 

2 2 

that its goals and objectives closely paralleled those of T S , the value \^ 
of SADT is confounded in the minds of most TRADOC officers. Xo paraphrase 
the comments of two TRADOC officers, SADT should have been applied to aV> 
new problem that had never been studied^ before with everyone working on the 
problem well-trained in SADT. ' 

Concerning the relative coi^tributions of each study to the 
achievement of insights into tank training, the consensus was that SADT had 
done little to isolate training requirements having a bearing on combat 
effectiveness and, battlefield readiness . The opinion of those TRADOC per- 
soniiel polled was that the^T S group actually did the work that led to 
conceptual insights with differing opinions about the relative contribution 
of SADT. 



Another theme which emerged from the qualitative data that could 
account for the skepticism cqncerning subsequent usage of SADT was that 

; V:w ' ^. ' ^ ■ 

of personnel turbulence. - SfivcJ^ral ^treapondenta mentioned this factor, as 

. '. ^ . 

a variable moderating the aecc^ptance, usefulness and ultimate success of 
a project such as this _one. Ope officer said that by next summer perhaps 
as many as. five of the top-ranking^^'6ff leers who were responsible for or 
participated in the SADT project may be gone. 

It is apparent, that' personnel turbulence in thie Army will effect 
most any project. , If the project is one that involves cooperation and fimei 
commitments from particular Army personnel, the efforts of the contractor 
are .particularly dependent on personnel changes within the Army. . The 
longer the length of the contract, the greater the probability of disruption 
to the ongoing work. ^The problem is not only one of trained Army personnel 
being lost to the contractor because of assignment changes but also one 
of acquiring the time of new officers for training and obtaining ^tlle needed 
commitment from a commanding officer who may also be new. 

A final theme that emerges as a possible obstacle to the success- 
ful implementation and usage of SADT was the relative difficulty in insti- 
tut ionalizing the technique. Most individuals polled believed that for 
SADT to^e used effectively it would have to be internalized. With few 
exceptions, the* view expressed was that SADT could be a valuable problem- 
solving technique ' ''brought to fruition within the Army.*' Opinions 
ILffered on how to internalize the technique but some general consistencies 
that emerged were: ; ^ 

V .• ' . 
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, (1) it takes bright individuals to learn how to 

autfior S^T diagrams, not everyone could be 
trair\gd; ■ 

(2) if u8£wby operations research pedple* then . 
, • . mana^ment would have learn an(i accept it 

•/ : in ord^r to cferamunicate with their problem- ^ 

solvers; ^ 

(P) ^ if accepted' and used by m^na'gement, it^wOg^ld 
filter* its way throughout the organization; 

, ' * • ' ' • .'^ 

(4) some abridged, simplified ftDrm that did not 
require such extensive training would have a 
better chance of being accepted and used; 

(5) if the technique clearly solves problems and/or 
saves time and manpower, and if individuals 

^ are sufficiently trained, the technique definitely 

will 'b^ used; 

(6) complete dependence on an outside contractor 
will hinder the acceptance of any technique. 

To summarize, although most respondents (13 out of 14) indic^tted 

SADT was a valuable technique at a general, context-free level, there was 

' it 

considerable skepticism about its subsequent applicability and usefulness 

at TRADOG. Several consistent themes that emerged from the qualitative 

data were presented as potential explan^^i^n:5-<f^^ the discrepancy between 
the context-free and context-specific views of SADT. Among these were:^ 

(1) insufficient resources being expended on the 
project; 

,1 ^ 

(2) lack of command emphasis on the project; 

;(3) lack of clarity about project goals; 

2 2 

(4) overlap between the SADT project _ and the T S group 
study; 

^ f 

(5) personnel turbulence in the Army; ^ 

ft 

^ (6) difficulties in institutionalizing a technique 
such as SADT, 



Discussion , ' • • 

The data collected in th6 Phase 1' evaluation indicated that 
officers within the TRADOC organization considered^SADT a powerful and • > 
sophisticated approach to systems analysis, design, and management, 
Qiiestionitaire items relevant to the context-free value of the technique 
were generally rated higher than other item6. Comments obtained in response 
to discussion questions and interview p^bes reinforced the concLuslpn that 

■ ^ - ■;■ ' . ' V ■ 

the .generic qualities of the diagrams themselves,* such as ^communicative 
quality, effic\iency, accuracy, consistency, and completeness, were present, 
as claimed in the SofTech proposal • " . * 

Ifhe same is true about SADT's heuristic ^properties . When 
questioned about the ^techrii^ue per se , mpst individuals indicat^ed a belief 

In SADT's ability to promote conceptual insights into problems. It. should 

V . ■ • 

be mentioned that individuals were somewhat less sure of SADT's heuristic 
qualities than they were about the generic qualities of the diagrams them-^/ 

selves. It can be concluded Vhat the sample for this evaluation study/ 

■ ' ■ * /' 

, ■ ■ ■ / 

valued the SADT methodology and generally believed the claims made by 
SofTech in the technical proposal. 

Queries about the value of SADT to tHe TRADOC qrganizat ion did 
not yield the same positive responses as did questions about the technique 
itself- It appears that some characteristics of the SADT/TRADOC interface 
moderated individual's opinions of SADT. BotK quantitative and . qualitative 
responses about SADT's applicability and usefulness within the TRADOC 
context ranged from mild optimism to open sk^ptici^^ The discrepancy 
between context-free and ^^cbntext-specif ic valuations of SADT led the 
evaluators to examine more": closely the interface between the technique 
and the environment- The process data collected provided a star-ting point 



.for -this examination, . / . 

The available process data "Indicated that SADT had not he^n 
implemented during the Phase' 1 evaluation to the degree prescribed in the 
technical^ proposal* The eonclusJUw::' <irawn from these data is that partici- 



patlon TRADOe personnel in the iterative ' author-tommenter cycle had 
■ ^ been, approximately half that expected in the usual application of SADT* 
The limited availability of TRADOC personnel for authoring SADT diagrams 
' P'^^r the fragmentation of the, TRADOC interface role during Task 2 further 
limited the impleraftttation of SAPT at TRADOC. 

■ r . 

The open-ended probes built into the evaluation design provided 
respondents with a **f ree-hand*' In expressing their views about the SADT 
project, and from these interpretive dat^, potential exftj^^natlons of the 
n...rginal implementation of the technique emerged.. The most consistently 
reported of these were that insufficient resources had been expended on the 
. proiect and that there had been a lack of command emphasis for the project. . 
^Possible explanations for these project weaknesses included a lack of 
c^.crity about project goals (which could be a function of the nature of c 
tlie problems addressed by this project) and personnel turbulence within V 
the Army, which contributes to a lack of continuity in management perspective. 

Another point that emerged which could be closely related to a 
lack of command emphasis was the difficulty encountered in attempting to 
institutionalize a technique such as SADT.\ If the project is not considered 
a particularly high priority by Army management, this is probably coramuni- 
Gated to subordinates in many ways, e.g^, 6fficially and/or unofficially; 
purposefully or unintentionally. If communicated to subordinates, difficulties 
will probably l?e encountered in establishing the procedural mechanics of 
t'--^ project. The coimnent of one officer that his participation in the SADT 

GJ 
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7ro,fi5Ct had no *b^arlng on his job as far as his. supei'ior was concerned 
llljus.tratcs this pblrtt. • ./ 

//> . The views of Army m^n'agfement did not actually refute what has • 

I' '. ■ . . ' ■ '. 

be'eti presented' thus far* These Individuals admitted that So^f^Tiech had not 

rt.celved the support expected and offered th6 rationalization that resource 

dcmandfii always exceed resource supply In the TRADOd environment.. One 

officer stated that this resulted In TRADOC being an unfair test bed for 

SADT. Another officer stated that such a hectic, complex environment was ^ 

an excellent arena In which to assess SADT's applicability to complex 
n . " 

. military problems. . . 

• . . / ' 

The appropriateness of the testing site not • withstanding, two 

. co^.cluslons are clear, from examination of all available sources of Phase 1* 

evalua on data: (1) the Implementation of the SADT methodology had been 

marginal, and (2) eight months Into the contract most TRADOC personnel 

involved In the project were unsure of the technique's applicability and 

• V ' ' 

uffefulness In the' TRADOC environment. 

In answer to the primary questions posed for the Phase 1 evalu- 
ation effort, the SADT methodology was considered ' to be a highly effective 
one generally^ but Its applicability and usefulness within the TRADOC envlrc 
•Tiont v;as moderated by several factors. Not the least of these factors were 
the organizational characteristics of the military, which predispose 
individuals to view a problem' one way, and the ii^der lying philosophy of 
SADT, which prescribes a set of behaviors for attacking a problem in a 
• ' ' ~ foront way. 



^ • . . ■ ■ 

I* ; . VI.\ Phase 2 Eval^uation Vroqedures and Results 

This section describes the ptocedurciS and results 6f the Phase 2 

'•J . . , ■■■ ■ 

^ - • : ■ ' " T ' ; i - 

^A«p.uatton. Uelhg thJ^iPhase A i^valuauion p^rotjedures and results^^as 



gUxflelines, the Phaise 2 evaluation effort was poncerned primarily with 
determining whetheir the attitudes of TRADOC pei^sonriel towards SofTech's 

work vftad changed as ,4 result' of -the productlonj of the Task 3 model,* 

' ' ■ ■ /- ' ■ ' ' ''I'. ^ ' 

Emphasis was glvriii In the Phase 2 evaluation tp; the perceived effectiveness 

and usefqln^tt of the Task 3 model, hoy this mcj^el was being used and what ^ 



impact t^^'' model was having on-^Army training j^nd' evaluation concepts, . 

This phase Served as a follow-up to the Phase 1 evaluation, qp- 
d^ing a^d 'documenting project activities and describing attitudes of TRADOC 
'^concerning the perceived eff^tiveneas and usefulness of SADT. Examination 
of current . proj ect activities v/as conducted to validate solicited opinions 
about SADT an4 to suggest potential indicators of the ways in which the 

SADT project may^ impact Army training and evalu£g:ion programs. 

■jf, ■ i-- • 

Methods and Procedures 

• , J. • » 

Nine individuals having close contact with the SADT project were, 
interviewed appro?timately one month after the completion of' the. Task 3 
.model or approximately fourteen months after the start of the project. This 
^roup consisted of five colonels, one It. colonel, two majors, and one 
captain. The conclusions drawn from' the interview data are presented . 
below in order of thipir consistency across the nine respondents. Special 
weight wa^ given to ;th'p comments of those ^having major responsibility for 



*Task 3 resiiil-ted in the generation of three models in the areas 
of evaluation, system| development-, and training. These will be referred 
to in this report as |:he Task -C^^^jsj^Qdel . 



this project. Appendix D contains representative stimulus questions used 
t.p promote discussion during the interview. 

The interviewer also completed a structured questionnaire based 
upon each respondent's Interview responses. The items for this structured 
questionnaire (which appears in Appendix C) were selected from among 
the relevant items on the Phase I questionnaire completed by thes.e 
respondents at the conclusion of Task 2. This permitted the comparison 
of, attitudes before and aftrr the complet'ion of the Task 3 model for a 
iuunber of respoiulents . 
Res^Lilts _ajul_ Discuss ion 

L'he most cunsLjitent themc- evident from the comments of respontlents 

wa^; tliat the Task 3 model h.ul made' a defini,te contribution in bringing 

f 

about needed changes Ln Army training and evaluation^:)rogram^. While 
r.vspontlfu t s vat t<'d iii how slguLf leant they th(nig,ht the So f Tech contr Lb,ut Ion 
wa^;, tt was rloai ^^H^t t lie 'i'KADOC t)fl'lcei-s polled valued the Task '\ motle 1 
and considered it an improvement ovei where TRADOC would have been at this 
po i nt: w i thout it. ^ 
Miv'.t respondents cmis i.th* red the major value of SofT^H'hV; work to 
'^o in dodimont i u^"; the in t o r rt* la t I oush l\is between evaluation, system 
dovoiopmrnt , and training';. I'oi" oxample, the Task \ motle I . dep I ct.s In i iirm— 
itivo t oedhack" as (he iuleiMact* lu*twc*cMi t ht* heie ti) ! \m o rndlvidually 
ronroi)Lua 1 i ::od ovaluat ton, j;vstom development, and t tMininy; comj:)i>nent s , 
fUu-: I i ak 1^)}', (hos«' liinr( ions in a s'y.stoinat i c mannoi'. Re^;pondouts indl rated 
(■lat i nd i V i diia^l s within TKADOl! mav iiave had a c ItMr t'on(*e|U ua 1 I /.a t loTv-^'f 
how [)aris ot ( lu' sv>'i( em i (M whlrli ^^hev wei.a' res»ponsihle opei'atod, bii^ 
the l\edv \ model prtu' Ided aiT i n t e^^, ra t otl plrtnie o\ (ho tola I Aiinv 
t raining; and ev<iluat [on sy^;t em and how that system muf;t Interact with 



combat developments. ThLs Lnslght apparently had not been achieved, or 
at least so vividly portrayed, previously, and is cited here as the 
probab,lc basis Cor considerable enthusiasm by TRiVDOC personnel f<^r actually 
beginning the work of implementing the new training and evaluation programs. 

The following comnients from interviewees are representative of 
tliis newly achieved integrated conception of evaluation, system design, 
and training: 

The SofTech project has been useful. The Army 
bureaucracy is filled with bright fellows v/orking 
in various places but until SofTech came thc;se 
^ ideas had not been put together. SADl' was a 

mechanism for bringing about interaction. The 
ideas were the rt^ a 11 along but had not been 
in te^; rated. 

No th L ng now appeared i n the models bu t they caused 
me to think of them (training issues) in a different^ 
light. The further delineation of what evaluation 
is has changed our concep t of evaluat ion and how i t 
Lilt iM" re 1 ciLl'S with the tLVilnlng system. . .Th Is year 
l>;i.'; seen the realization of what we've been (talking) 
.iht>u t. lor three yea im.. 

The iiiscussi.on with Sofl'ech imd examination of 
diagrams was one of several Int.el lectual aetivities 
that defined and ch'^.c r i.bed where we were heading In 
TKADOC. The Army hasn't taken the process' but the" 
products; (>r that; pi'et:es:; are valuable in whete we're 
lieatl i ng, . ' ' 

Si) I Tee h 's wi> rk lias ma tie peop le think. It t i e^; th lugs 
loj',ether. SolTeeh's work at: '['KAOOC lias provided 
oilman i /.at ion t TRADOC itst^lf; it has provitled a 
seiist' oi d i ree t i iMi t.o t he wlu) 1 e i'onet-p 1. of TRAHOC' . 
1 t ha;; f oree'tl TRADOt) Lt> a c 1 ear eoiuH'p tua 1 i v,i\L ion oi' 
what t hev * re do 1 \\}\ , what t hey 1 iit end and what; tliev 
want io 1 till' I ut ni'i' . 

In a c U I i I I o n , a r t > 1 o u t - 1 1 1 a v i u ^ s i j\n i f i e a 1 1 1 r e : a u >^ s i h I I I t y tor the 

.if ' ' l Ot li jM n j ee t i lid i a I eil : 

l''u> I stMUt' I i me we (lioiu'.ht we undtMstixnl illtlerent as[>eets 
ot the sv.'iteni, but loi the t i.rst t line we have a elea»' 



picture of the interrelationships of different 
functions and organizations of the Army as they 
relate to training. The model systematized and 
coalesced aur knowledge. ^ 

Between the Phase 1 evaluation report and the present foLlow-up 

conducted four months later, a marked shift in TRADOC's attitude toward 

SADT and Its products was nested. This shift was traced to the emergence 

of the Task 3 model and/or the activity which surrounded its devel9pment . 

iMost oi' the skepcicism reported by Tl^ADOC personnel in the Phase 1 report 

pertained to the usefuLnciSB of SofTech's work and the ultimate impact it 

j 

'/t)uhl have on Army tralnlnj!. and evaluation procedures. At the end of Task 3, 
this skeptic ism had been rep laced by enthusiasm abou^vthe use fulness of 
Siill't^ch's efrcn'ts. The Task 1 niotlol seemed to have brtnight together and 
solidified many the Ideas generated by TRADOC , but that heretofore were 
seen as disjointed and insufficiently articulated to be of pract ical use . 

Th(^ com: I n.^M ons are supported hy results from the structured question- 
iiaiT'o. StweiMl i t onis had hij'JuM" (more positive) mean rrsptnises afttM'-Task ] 
than they tl Itl at the conc:luslon of Task 1!. These comparisons are presented 
in i'iy;nru 5. 

I'ivc^ iLcmj; evi^denced rrjat lvely large changej; In mean response across 
the two .'lamp 1 i iij;.'5 . The laI"^*e^lt. ch.inj-»o about l.S scale nnits) ocu^iu'red 

witli i\*j;pect to t lie rcj^ponchui t * s undtM^s t aiul Inp, ot Army traininj* and evaluation 
priM^raiiLy. ( Lt tMii S>. Ot her chani*e'* ^Mi the orilt^' ni' \ sea nn i t can be noted 
t oi" item , (ndicat i nj^ that. re.spnndiMi t.r. were ut^w mr)re confident tliat^SAOT 
diar.r.nns i()ulil iiicreaso Tl\AI)()(lV'; planninj',, thvsly»n «iud inana^»c'"it'u t capahiMties 
.itui tor i t riu:; <i , /, aiid IS indirat i n^'. t iiat rivipoiulent now saw more n;;e:i 
tor tMie mt)del.s prDihuctl, '^'iw the dia^»,rams as i\ioi*e instrumental in es t ali I i -di I ng 
training', lecpi i i omeut ^; » and piorcivod ^;reatcM- impart ol SADT on t lu* training 
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concepts currently held by TRADOC. Changes between .75 and 1 unit 
occurred on items 2, 10, and 17, suggesting that 'the production of the * 
Task 3 model had stimulate'S'.^more communication among individuals not 
directly involved in the project, that TRADOC personnel now considered 
the SADT models produced more valuable as. reference documents than they 
did at the conclusi on of Task 2 and that TRADOC per sonne 1 were now more 
convinced that the resources devoted to this project tiad «been well-spent. ^ 
TheYPhase 1 evaluation indicated that the degree of implementation 
"of the SADT methodology- had been marginal, particulm ly tho training ol ''^ 
SADT authors, and suggested that lack of implementation might limit the 
utility and impact of Sof Tech's work. The skepticism about the value of 
the work produced through Task 2 was attributed earlier to this lack of 
implementation. It may bo, howe^^ that the content of the Task 1 and 
Task 2 modelr; Limited the sj|j^l^^^-s that could have been made about ulti- 
mate usage ami impat I . Huk/V the' actual production of the elaborated model 
of Army training and evnLuatiou resulting from Task 3 may have been a 
necessary prerequisite to TRADOC officers verbalizing any specific benefits 
of" the SAlvr project. Further, posLtivt^ valuations of So f Tech's work at the 
end ot' Task i sugj;estel1 that the degree of participation by TRADOC in . the 
SADT proees.s hail been suf f Leient , a Ltlioagh less than originally expected. 

tasks I and 2 reprt\sentetl necessary Init prel Iminary steps tliat 
prepared the SADT allalyst!? for the task 3 effort. One officer stated tluit i 
SotTeeh "etit tlun.r ltu»th" on T<i!;ki; I and 2. During, this t Line , officers 
had .1 dill ieult I iim> seijlng, I lu' value t>t SofTeeli's work. Uee<uis,e no one 
eouKl arl leulale wh.il a model of ihr Army training. systiMii sliouhl look 1 Lke 
betore aetual pro^hiel iou ot that model, 'tKAlH^C oTlieers may have Iuhmi 
uneiTt a i n to liow t ho 1^isk I ami 2 results would provide^ a better 

(.(> 
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understanding of Army training and evaluation programs. This also suggests 
that project benefits to TRADOC are directly attributable to the tangible 
outputs of the SADT process. 

In summary, attitudes toward SofTech's efforts at the conclusion 
of Task 3 wore positive. This represents a shift from generally neutral 
or skeptical attitudes expressed by j TRADOC personnel interviewed at the 
end of Task 2. The relationship between training, evaluation, and system 
development was perceived by TRADOC personnel tq^Jti especially clear at 
^the completion of the Task 3 model, while the limited focus of Tasks 1 
and 2 may have contributed to the skepticism and ambiguity noted in the 
Phase 1 evaluation. ^ 

Although there was a high degree of enthusiasm at TRADOC about the 
new conceptualization of Army training and evaluation, it would be pre- 
mature to concl.ude that this result was solely attributable to ^ the pro- 
(hu'tion ot t ho Task i nioch* 1 . Ot her I notor^? oont r Ihut ing tcv t his outcome 
may liave inciudod the part Leu La r insights (and foresights) of key TRADOC 
officers, the act-ivities of other projects at TRADOC subtle organiza- 
L lona L charaeto.r LsL Ics which may have evolved at TRADOC, and the ze L^ei.^t 
currently present in TRADOC anil Lu the Army. While this list is specu- 
lative and, far from inclusive, it i offered to" hulieate that TRADOC'-". 
current uadersLaiul tur, ot tralainj; and evaluation may he the result ot 
the r<Mil huMice of a large number of varlahlet;, many of which could not 
he mea.sured given llie time and rescniLyes allotted to this s.t.udy. - Further- 
more, |fhese pt)Lent i.il iiifiiuMices may interact: witli each other In complex: 
wa^'s result in)\ In vmiciue comb lyat;or i a 1 eflt\cts. 



Thus» a potentially large array of Influences may have interacted 
to produce the attitudes measured in the Phase 2 evaluation. The opinion 
oxnro.ssed by most TRADOC personnel was that SofTech ' s Input was a part 
of this larger confluence. Two officers, how'ever, exprffesed that SADT 
brought organization to TRADOC, that it created a different climate at 
TllAVlC, influenced other projects, etc. This would su|:^gest^ that this 
combine of influences falls within the influence of SADT itself. The 
)'oi^re:^eat:a t ions lit Fi^,^iiro 6 portray several of the conditions that may 
h.ivo existod at TRADOG lo procjueo the Ns^titudes measured during the 
Phase 2 ovaluatiuu. ^ 

Figure 6 

■ .} Combination of ail influences 

affecting TRADOCVs perception of Army training 




SAirr is amon^; nianv SADT is the nn\y SADT is the. i)rlmary 

f ai' ( o r^; int I lUMir i \\\\ far tor I n f I uenc i nr. fact or 1 n f luenc ing 

TKAOOC's porci^ption TRAUOC's piM-cepl ion TRADOC's perception 

>• Armv (raininf. i ol Army training of Army Lraining. 



Which ti| t lu'sr rt*l\il i^n^;llips iih.>st aoriiratolv portr.iyr. t ho role* oi' SADT 
(n Tostorin)'. ch.iiij'.t' at TKAHOil lU'adcjua r t r s cannot b.r doLoriuinod by t lu' 
prosi'ut dalvi. i'h i s d ( srnss i tui , Ihi^i, is prosoiU rd .is a rani ion to t host* 



who would interpret the Phase 2 evaluation as indicating that the 

.J ' ' ■ > 

production of the Task 3 model was the sole causal agent in bringing 

about changes in Army training and evaluation. 

That TRADOC was pleased with the product of the project at the 

conclusion of. Task 3 was evident. The following comment from a TRADOC 

colonel summarizes what TRADOC received from this contract through 

Tar\ 3: 

The principle value (of this project) has 
been the identification Of sources of infor- V 
matlon relative to how the system ife working 
and feeding that information back into the 
front end (system development). (This) will 
increase our ability to better target resources 
we put into training and hardware development. 



The contract proposal listed three specif ic benefits that would ^ 
be derived from the Task 3 model: 

(1) . hetrer understanding In TRADOC of what it Is try'ing to 

cuhitwe; 

(2) case of telling others, the schools, and Department of 
the Army what is going to happen; and 

(1) recUutiou of t. i mo iu planning, managLilg, and accomplishing 
the changes . (Contract proposal p. 3-LO) 

At>«Lhe .one Ins ion of the Phase 2 evaluation these benefits had 

ixHMir red 
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VII f Phase 3 Evaluation Procedures and Rosults 

General Approach i 

This section describes the procedures^ and results of the Phase 3 
evaluation. Building 'on the '^results of the Phase 1 and Phase 2 evaluations. 
Phase 3 determined post-project opinions of TRADOC personnel towards SADT . 

The focus of Phase 3 differed somewhat from that which was 
originally planned. Because this change necessitated a shift in evaluation 
sLrategy, a discussion of the rationale for this change follows. y^^-*^-^ 

Although the Task 3 report was well-accepted at TRADOC Headquarters, 
the Task' 3 models had received little exposure at the 21 TRADOC schools 
at the conclusion of Task 3. The models were considered reasonably-"specif ic 
by personnel at TRADOC Headquarters but the viewpoint of individuals at the 
V RADOC, schoo Is was that the models were abstract and of questionable .practical 
v<ilue. Further, t:he mode Is emphat^i^.ed new roles for the srhoo.],* involving 
lioavy usago of oxportabLo training, assigning various schools the responsi- 
Dility for \7eapon system perform^mce and requiring schools to use performance- 
o r i <f^u tod , c r i to r N )a- referenced measures of training effectiveness. The 
LraditLonal role of TRADOC schools (trralnlng resident students) was 
iU*tMup(ras L/.od. I n^i^he Task 3 model. Foll^)wing disseminatiou of the Task 3 
roiH^rt it dill not >ippoar l ikely that TRADOC school porsonnel would readily 
adopt, t he mow model of training because of its lack^-trf spec I f Ics about 
oat h srhool^j wo.ipoa syj:;tem and MOSs and the emphas l-s on newer rather 
than luoro traditional role:; for TRADOC scliools. 

It wa.s oviilont 1 1> SofTorh and TUADOC personnel at the conclusion 
, Tar.k \ that ai'iu^pt ani*o ot tin* new couiu^ptua 1 h'.at I oa i)f Army training 
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and evaluation at the various proponent schools was critical to the 
successful implementation of the model'. Further, becciuse of the extensive- 
ness o fa the Task 3 model, it was believed that emphasis should be concen- 
trated on TRADOC*s most immediate critical needs rather than the broad 
focus of developing an implementation plan for the entire Task 3 model. 

^For these reasons, the decision was made to drop the implementation 
plan which was the original focus of Task 4. Insttead, effort was to be 
devoted in Task 4 (to several activities^ including dissemination of the new 
model uC Army training to the proponent schools of TRADOC to foster 
acceptance of the model, development of a plan for implementing some specific 
evaluation feedback loops into the existing training structure, and 
beginning a plan Cor a systcMii architecture for the data processing support' 
requirements ^ of the new model oC training. 

Because the new focus of Task 4, did not emphasize the use of SADT , 
the Task 4 acLivit Lcs were not as directly relevant to the purpose of the 

V 

overall evaluation — the evaluation of SADT in mocieling a military training 
environment. Further, t iicse and other project activities subsequent to 
Task 3 did not {H'(nn: early (MU)Ugh to allow for an analysis prior to the 
preparation dI this re[KH^t. For these reasons the Phase 3 evaluation focused 
on a finer deterininat um of TRADOC ' s percept Ions of the eT f ectiveness and 
usefulness of the SADT pro)tM'L, [la r t i cu lar ly the Task 3 model and specifically 
the factors whirh may have been responsible for the positive shift in attitude 
which ocruri\eil between the Phase I and Phase 2 evaluations.. 
Methods and PYoetuhif es ^ 

....... . q r . 

'y\\r data <v>lleetien piMMod for the Phase 3 evaluation occurred 
(hirtnj'. duly and e.irly Auj'.us, t M> 7 / . Tlie pers.onal va(\il fen sidunlules 



and professional obligations of TRADOC personnel during this time period 
mdcie it impractical to attempt to interview relevant TRA.DOC officers in 
depth about their perceptions of the SADT project and the Task 3 model. 
For these reasons a mail-out questionnaire, similar to the one used in 
the Phase 1 evaluation , was selected as the primary source for 

Phase 3. This questionnaire appears in Appendix D. 

Fifteen TRADOC officers having raanagemerftr^;.^esponsibility for or 
pa*rt icipating in the SADT project were selected as the target population 
for the ques t Ldmialre . Two weeks after the questionnaire was mailed, 
an evaluation staff member went to Ft. Monroe to stimulate the return \ 
rate of questionnaires and to interview as many of the target population 
as possible about their overall impressions of the project. Questionnaire 
and LntervLcw results are presented and discussed below. * >' 

Uosu 1 t.s ixnd DLst'Us's 1 on 

Twelve oi the fLLttUMi ques t Lonua 1 l\\s were returned, a return 
rate ol 80%. Further-^ lour individuals were interviewed during a final 
visit to TRADOC Headquar terr> by an evaluation staff member. The results 
ol thtise data p.atliering activities yielded two types of data, quantitative 
(fii^m Questions 1 through 21 of the questionnaire, Appendix E) and 
(|ua 1 i L.it Lvo (from (juestLoiis 22 throu^U^ 27 of the questionn laro and 
the i 11 tor views) . 

t Ji I a iltj t_: i_t^i vy. Dj it a . A r o 1. i a b i. 1 i t v c o e f f i. c :i e n t w a s calculate d 
tor tho ^^1 quaiU itaL i.vo i t(Mii.s on t-ho cjuos L i oaua I re . An alpha coefficient 
ol ,^)\ w.xry t)l)laino(l iiidioat liij; t:liat que5it i (Mina Iro. item.'; wer(^ homogene.ous . 
That i.s, tiu^y prodnced .simi lar pat t.tM-ns tM" rosponduig in different indi- 
vl<lnah;, iiid i oat i nj; the (|Ut\-',t i oiuia i re Wi\i\ lU'lial^le. 



The mean (X) and standard deviation (SD) for each item 
are presented in Table 4. ' 

Inspection .of the general int^ent of each item suggests that items 
can be clustered into relatively homogeneous groups. For example, Items 
3, 4, 7, 9, 13, and 14 inquire about various heuristic aspects of SADT 
diagrams and the modeling process, ' Inspection of the results of these 
questions suggests most notably that the SADT models produced were helpful 
in elucidating the interrelationships between the various components of 
training (Table 4, Item 4) and that the Task 3 model provided a clearer 
understanding of the Army's training and evaluation programs (Item 7). 
Responses to Items 13 and 14 suggested that the technique was somewhat 
less effective in identifying organizational inef f icienclea in the 
current training system or in providing conceptual insights about alterna- 
tive methods of conducting Army training. None of the questions addressing 
heuristic properties of the technique rt^ceived unfavorable ratings 
(I.e., a moan less vluin J . 00 on the five-point scale), 

, Items 8, 12, and 15 address the ul timate . usag^ andimpact of 

the models produced. While respondents believed the models would bo 
ui;ed to a considerable extent in planning needed changes in Army training 
(Item 8), they were relat ively less sure that the pipdels wpuld actually 

t would ult i- 

mately affect the training; concepts held by TRADOC. As with the previous 
Rroupin^^ of . Items, none of these questions received unfavorable ratings. 
\ Several items on the ciuc'.st i onnaire (10, 11^, 16, 17,. 19, 20 and 

21) ai;ked the re.spondtMit for his ^inmluary jud^^ments about various aspects 
of tiie SAD'l projeet. The most: ncUahle of these was that item 11 received 
the lii^;hesL rat luf^^; o\ any item on the (|uest ionna i le (4,33), indicating 
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provide tht^ Impetuj; for ehanj^,o (Item 12) or that proje 



Table 4 

Item Means and "Standard Deviations for Phase 3 Evaluation Respondents 

J 



Item 



Mean 



S tandard 
, Deviation 



-^^ — 



1. Are SADT diagrams an effective 
way of communicating? 

2. Has SADT project generated 
communication among TRADOC 
personnel? 

3. Have diagrams helped focus^ 
attention on variables not 
considered previously? 



12 



12 



1'2 



A. 33 



3.7> 



3.67 



0.65 



0.87 



1.23 



Did models elucidate inter- 
relationships between the 
components of training? 



4.25 



0.75 



5. Project benefits attributable 
to diagrams or process? 



11 



3.00 



0.8$ 



lias SADT saved time relative 
to other approaches? 



10 



2.80 



1.32 



Did Task 3 model provide 
clearer understanding of 
Army training? 



II 



4.00 



0.89 



Will the mod (Mr. pfodiiciHl hv 
used In p lann Inj; c*hanj;os? 



lias sad/" lelp ed identil'y 
changes ^^i t'ec. t Ing combat 
rrad i uos;;? 



1 2 



12 



3 . 50 



3.50 



0.80 



1.45 



10. Would aa SAD't modol ol Army 
training be of va hu* as a 
re iL^reuiM* cloc.iiinent ? 



4.33 



0.89 



11. Is SADT an etiecLive approach 

to cU'sign and analysis problems? 

1 . Will SADT mudols providp im[)elus 
tor ('han)',os In Arniv Iraininj;? 



II 



12 



4.55 



3.17 



0.69 



1 .03 



13, Wort' S.ADT moclob; ust*lul in ich'u- 
t-it.v ii^'. (*xist tiij» or^'.aii i ::a t ioiial 



i:> 



3.17 



0.94 



Table 4 (continued> 



Item 



Moan 



Standard 
Deviation 



14. 



15. 



Had SADT led to corip^ptual 
Insights about conducting 
Army training? ^ 

Will use of SAJ^.T ultiTuately 
affect trainiitg ^oncepts 
held by TRAIiOC? ; 



11 



12 



3.45 



3\25 



1.21 



0.7^ 



16. 



17. 



i9. 



20. 



Would It bo useful to train 
some Arfhy personnel to be 
SADT authors? 



21 



the pro juct hicreast^'d 
"^RADOC^s analyti^/ and 
1^1 ami in g capab'llit-ies? 

Uas the proit',ressOTfade been^ 
worth tfi(? ^t.\me ?^m^t? 

Kstimc^te of ^SofTech Vs ^■ 
contribution to the ']^ro^re^^s 
m|de^hy TRADOG-., 

'Task "] nipdel .fu iiuprt)vonRMiL 
over previmui roiu-cpt ua 1 i ^ 
/. a L i o n .s o t" ^'i^t 4* a L ii i. n ) 



-12 



12 



12 



12 
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3.17 
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4.36 



1.15 



0.83 



0,79 



0.79 



1 , 12 



3.67 



"0,55 



a HLroiij;; bol Lcf /itiuMig TRAI)^/perHoniu^l that SADT in aw of ltHiLLvti and apprc 
priate approach to the dcBlgn and analysis problems faced by TRADOC. Tlicse 

individuals considered the Task 3 model an improvement over previous con- 
ceptualizations of training (Item. 21) and were of^he opinion that the 
progress ihade was worth t;he re3our(;^es <ievoted to th^ project fltera. 19) . 
Further, these kidivldua Is considered the model to be a valuable future 
rc-Ierence document for TRADOC personnel (Item 10). 

Despite' the high ratings given to most itenjs, Question 17 was 
marked relatively, Liiwer (X = 3,17) suggesting that q limited degree of"r 
teclmoLogy transfer Irad occurredl This is congruent- with conclusions 

; ^ ' ^ . ' . ' 

drawn from the process tlata Ln tlie Phase 1. evaluation Where Lt was 
reported that- fewer tiian cfxpec ted - TRADOC personnel we're trained in 
SADT and Lessi thanTuLL participation among TRADOC ^ personnel was recorded 
lor the auth(j r/ijommeiiter Review cycle. Item 20 also receive^ relatively 
low ratings-(X.^ 1,08), suggesting that yhile SofTegh made a moderatew 

contribution ^to llie progress acliL^ycd by TRADOC, it was not a major 

* * - ' ',- ■ ' 

(^ontr ibutai;- , This mi^ght be expected from the interaction ^of Sl^iDT and 
otihe^r influences on Army progress discussed at the conctusion of the 
Pha'se 2 evaluation results ^Cpp^ * o f .th is repo^^. 



( j u a I i t a t i. V e_ Da t;a ■ MWe ([uestionnaire contained six open-ended' 
q-'-r.t ions ( Append Ix 'D , '(;ijii,Qstions 22, 23, 24, 25, 26, and 27). One of 



these ^ ( 2(y) wii 1 b(^' d iscusseu'^'ia a, subsequent section of thi^^ report'. 
Qut-stion's 22, 23,^24, and 25 addressed specific issVies that were eithor ^ 
^left. unclear o/ emerged from tlie Phase I add 2 cvainat ions . ■' 

A conclusion drawn from the Phase . 2 ' evaluat Ion was' 'that ' 
ar -'i-tirdes at TRADOC toward SADT had shl f ted « pos i t ively following 



...i 

dlascmination of the Tatik 3 report:. Several tentative expranation« for 
this shift were offered at the conclusion of the Phase 2 evaluation , 
report. The opinions of TRADOC personnel were solicited to further specify , 
why this shift had occurred (Question 22) , The most consistent response 
to this question was that Tasks 1 and 2 were necessary preliminary 
activities of limited scope while the Task 3 model constituted the "meat" 
of the project. The following comments by four officers are representative: 

^ Since Task 2 dealt with a specific weapons system 
* (the tank), *many officers were either unfamiliar 

or uninterested in the Task 2 report. The general 
subject of the training system is more -widely 
understood and generated ia greater degree of interest. 
The meat of the contract .was in Task 3 and Tasks 1 
and 2 were merely, preliminaries. 

Task 3 seemed to' come to grips better with the 
key Issues. * After all, it represented a. high 
point in the Learning curve. Simultaneously 
it occurred as we were tryiwg to deal with ' , 

(other agencies) on total system developments. 
Httncc, it.s insights wore of benefit in the 
dally battle. 

The Task 2 report was not promulgated as a ^ 
solution. It was a me^ns to an end. The Task 3 
report', on the other hand; was more general 
and intended for widet distrib^ution. Thus,* 
improvement in opinion should occur . 
t ■ > 

IVisk 3 showed the /'big picture" f or the 
first time. People could see how their 
pieco, fit in and where the gaps were. ' . 

(Jaestion attempted to assess whether the SADT project 

» ■ , . . 

, had iiad an impac t on TRADOC 's abil. Lty to solve large and/or complex problems. 

o 

Willie this was^otio ol Llie major benel'its implied in the "contract proposal. 
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the iaaue. had not been addressed directly in the Pliasn I aa<l f'hase 2 
evaiiiat Jona. The generai response to this question was that the 
\i. iividual officers directly involved in the project had probably honed 
their personal problem-soiving abilities but that SADT had had an 

organLzation-wide impact j^n analysis and planning skills. ^ It should )?e" 
mentioned that several respondents believed thclt a lack of command ^ 
emphasis (jurying the project may have limited the impact the project 
couLd be expected to have on these skills. 

A major conclusion of the Phase 2 evaluation i;:eport was that 

- / " ■ ■ ' ■ ' ■ •'■l> . V 

fol/LowLng dissemination of the ^sk 3 report, individuals were more 
coFifLdent of the usefulness of the SADT project and were more positive 
that ^his Wotk would ultimately be of benefit to TRADOG . However, it 
was unclear from the Phase '2 data^^$ust how the Task 3 model would be 
ii3cd or what specific benefit's have or were expected to accrue at 

tpj\doc; 

Question 2 3 inquired directly about ways [the Task 5 model would 
he used by TRADOC. Emphasis was given to the responses of higher ranking 



;iesponses to t^his 



officers with training management responsibility. 

■ /) ■ . 

quostioo fall into Lhree general c^ttegories^^ 

The mo^;t consistently mentioned usaj^e of the model wa^ as a 
cotnmunicat ions tool. One off^icer stated that\the model Would t)e used 
to "clarify the DA (Departmient of the Army) staff's understanding of. 
tlic t ra in ing'^system. " Another officer stated; that al major use of the 
modt'l was "to cxplaiji the pervasiveness of TRAIJOC's functions and missions 
to DA, DoD, and other major commands." Other responses suggested Task 3's 
usefulness as a communications tool within TRADOC , sti^essing that the 



modtil can be used to (ixplal.n to 7RAD0C personnel the intcrrelationBhlps' 
between TRADOC and other commands and agencies within the Army and the 
Opu. It is cleTir that a major usage of Task 3 Is as '*a way to e:^plain 
the training system.'* 

' Another consistently noted usage of the Task 3 model was as an, 
aid in the systematization and coordination of combat development and . 
tiaiaing development activities. ThiS'Was discussed as a major conclusion 
of the Phase 2 evaluation and will not elaborated here. It should 
suffice to say that the role of the Task 3 model ai^^i blueprint for the 
coordination of commit; development efforts (priftarily the development 
of specil'ic weapai^s systems) and training developments . efforts continues 
as one of the most valued of "the potential uses of the Task 3 model. 

A third category of use^ centers around the value of the Task 3 
'porlel as an analytical tool in such areas as the plannitig of management 
Information requirements for the training system and determining the 
system architecture necessary to support future data "^i^rocessxng needs. 

^'^ ' ■ ■ . ■ ' ■ 

A !cliLLonal .usages within this area include providing management a 
'perspective for assigning responsibility for various aspects of the ' 
L: Uniiig and evaluat ion. system and establishing criteria for the alloca- 
L Lon of training resources. ' ^. • 

j At the conclusion of Task 3, most respondents interviewed for 
the! Phase 2 evaluation report expressed that it was too early to specify 
benefits that might accrue to TRADOC as a result o*E carrying out this 
project. The Phase 3 evaluati(fc)n questionnaire followed these ■ interviews 
by approximately three months, allowing time for the model to "sink in" 



aiul for potential bene tits of the mode I to evidence themselves. As 
with the uses of the Task 3 model discussed above, several specific 
benefits were articulated by respondents in response to Question 25. 

A primary benefit listed by almost all respondents was that the 
SofTech project had resulted in the articulation of the Army training 
and evaluation system ^^err^t he first time.. Thl^ documentation was seen 
ns very valuable by TRADOC personnel Va^d is related to their ^p^c^ived ^^^^ 
ihsage of the Task 3 model as a valuable communications tool. Another 
often cited benefit was the. elucidation of the relationship between 
combat developments and training developnients . A third consistently 
occurring response to Question 25 Was that "the role of evaluation and 
feedback in the training system had been clearly identified and raised 
t) the desired level of prominence." 

' Other, more idiosyncratic benefits listed in response to Item 25, 

wore that because SADT highlights system constraints, required "fixes" 
tc present training system problems could now be more easij.^ identified. 
And, because the Task 3 model '"highlighted areas where automation was 
n. ^ded," a^tarti^g point for planning future harc||ware/sof tware needs 
l-ouLcl he determined. ^ 

A* d Liferent perspective was offered by one officer who listed 
LliL^ only project benefit as the "crossf ertilization or ide^s." He ex- 
nl,i''ied his response this way: 



(TlVe project) forced TRADOC to explain the > 
^ * system used to outs Ldc agents . ^JThis is a 

remarkyihly useful exercise for'^smug persons 
wHo tjiink thoy understand what -they do. 



..V 
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Given the cvnluator's experience with SADT and TRADOC, this comment aeemfi 
insightful and nondef ensive , and appears to underscore the mechailica of 
how SAJ)T accomplishes its results. 

, The idiosyncratic comments of another ^TRADOC officer deserve 

I, . • ' 

tnention. While acknowledging the potential benefits of SADT, he had 
reservations about whether any benefits would actually be realized* His ' 
response to Item 25 follows: 

As a minimum, TRADOC 'has been exposed to an 
enlightened form of analysis. The results ' 
now exist in a highly usable form. I am con- 
cerned about TRADOC 's ability to apply' the 
Task 3 model. This concern is based upon our 
lack of ej^perience in conducting such a massive 
project — the type of project fPjfcolicy-oriented 
staff is not equipped to handle^ 

Thus, the benefits of the model as documentation of what TRADOC 
Is trying to achieve is clear in the minds of mos^^^RADOC officers! Even 
the process of working through the models was considered to be of benefit 
tn the 'organization in and of itself. Further, the model app^ears to have 
considerable potential value as TRADOC Headquarters move^ to make needed \ 
changes in the current Army training and evaluation system. There is 
some skepticism, however, that TRADOC will be able to use the model opti- 
mally. As stated by one officer **the value of SADT is still more 'potential' 
than realized." 

Summary and discussion ^ The .mos t- notable trend from the Phase 3 
evaluation' is tjiat the attitudes of TRADOC personnel towards the Task 3 
model remain positive'. No deterioration of the positive valuation of 
the SADT work was.;evident three months after jdissemination of the 



il 't\ 



Task 3 report. Further, it appears that thij5 three month interval pro- 
vided an "incubation period" in which the results of' the Ta^k 3 work 
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had a chanac to "sink in" in the minds of'TRADOC per.sonnel. Phase 3 
GrVaiuatlou results indicated the model had not been put on a shelf 
and forgotten but rather had become a central document in rpi\J)0C'8 ^ 
thinking about training and evaluation. The high return fate for the 
Phase 3 cjue?ytioiinairG% and 'the thoroughness with which most respondents 
answered, items on the questionnaire seems to support this conclusion. - ^ 

The conclusdfcs drawn from the Phase 3 evaluation data were 
similar to those in Phase ^2, namely that TRADOC is satisfied with the 
results and^tonsiderecf its investment in the project worthwhile. .The 
general opinion expressed was that the groundwork had been' laid to con- 
^vert the Army training system from what it is to what it should be, 
Wtietlier the potential benefit^ of the SofTech project are realized is 
dependent on two factors in the minds vof- ttiost TRADOC personnel. The 
first concerns the emphasis given to the model as a planning tool by 
apper level nfanagement at TRADOC. Throughout all three evaluation 
phases, this has been an expressed concern of officers involved in the 
project. The 'ultimate -use (and derived benefit) of the model is depen- 
C.oMt on the priority given it by senior level managers in TRADOC, The 
second factor concerns reservations about the ability of TRADOC (more 
specM' Cically ODCST) to use the results of the Task 3 model in working 
toward the state of - training and ''evaluation desired by TRADOC. ^ 

The interrelationships between the results of the three eval- 
viaL : . phases and a*finai project-wide evaluation summary follows. 



J 



Vril. Project RosuiLs and Discussion . ' 

• " * 4 

In this section trends across the three evaluation phases are ^ 

integrated and disNsussed. . 

Process Data J ' 

Considerable emphasis during the Phase 1 revaluation-, was placed 
on the SADT process. With the exception'of one TRADOC officer ..trained 
as an iSADT author, conjienting on and approving SADT kits represented TRADOC's 
only official contact with SADT. Thus, any benefits derived by^ the client 
organization would Hcem depe^ident on completion of these process behaviors. - 
It was for this reason that the process data from the project file was 
examined' during Tasks 1 and 2. * „ 



The Phase 1 evaluation repx^rt indicated^ thai: TRADOC ' s participa- 
tion in the process of diagram review and :^^ision was about 50% of that ^ 
originally expected by SofTech. Despite this result, none of , the project 
personnel polled at the conclusion of Task 2 believed that deficiencies 
perceived in the proj,ect at that time were attributable to this reduced 
degree of participation in the iterative author-commen^er cycle . Most 
respondents indicated> that ample^ communication had occurred between SADT 
authoFs and TRADOC commciiters and that, if anything, the formality of the 
3ADT process had hindered TRADOC's participation in the project to some 
extent. ' 

•■■ ^ ' 

Prior to Gompletion of Task 2, SofTech altered the SADT process 

' -^-^ ' ' ■ ' . 

slightl^'*to achieve a better match between the process and work procedures 
In. the TRADOC organization. ^Primarily., co^udicat ion ' within the SADT. 



process was expanded to inciu'de conferences and open discussion of the 
, status of various, aspects of the mode]|. Because both SofTfech and TRADOC 
expressed satisfaction with the levfel of communication between authors and 
commenters and because th^ procedural aspects of SADT had been altered 
to meet the needs of this particular application of the technique, the 

issue of process data and imolementation was not pursued fti subsequent 
' evaluative phases. It appears that the traditional procedural mechanisms 
nf SADT neied not be strictly adhered to ^f appropriate alteij^tions can be 
made to facilitate participation in the process by personnel in the client 
organization. fhe^ fact tliat positive project benefits were derived de- 
spite less than strict adhiurence to traditional procedures supports this 



contention,' ^ A 

, \, . ^ - ) 

.rerceiv ecTlMSE gctiveness and Usefulness of SADT . 



Ex^ination of the res^ults of the three Valuation phases 



indicated that a favorable change in attitudes towards SADT by TRADOC 
personnel had occurred. The Phase 1 evaluation concluded that the 
effectiveness and usefulness of SADT had not been clearly demonstrated 
at the conclu^,iort of Task 2 and that the respondents indicated they 
believed the technique was a good one, generally, but had reservations 
about the ultimate impact of its application at TRADOC. 

The Phase 2 evaluation, which followed Task 3, concluded that 
attitudes towards the SADT project and its results were positive and had 
changed considerably frojfi what Was found during Phase 1. • The tost con- 
sistently cited reason tor the more positive sentiments found In the 



Ph-ise 2 (Evaluation was that the Task 3 model had provided an Integrated 



4 



picture of the totcil Army training anc| evaluation system, particularly 
the interrelationships between evaluation, system development, and 
training. This integration had not been as vividly portrayed previously 
and generated considerable enthusiasm for beginning the actual work of 
implementing the new training Snd evaluation system, 

Phase^3 evaluation results were similar to those found in 
Phase 2.- Attitudes towards the SADT project remained "positive. Despite 
the obstacles encountered in beginning the actual implementation oS 

' • \ . 

the new 'training and evaluation system, the Task 3 model was still con- 
sidered valuable documentation of TRADOC's best thinking on how the 
Army's new training and evaluation system should operate. Most respon- 
dents were of the opinion that the project has been a' worthwhile experienc 
for TRADOC and expressed that the groundwork had been laid for beginning 
actual implementation of the new system. ^ 

Thus, TRADOC personnel were originally unsure or ^en/skeptical 
about the value of SADT but attitudes towards the technique shifted as the 
project progressed. Only t%o TRADOC officer! questioned during the 
Phase 3 evaluation stated th|t they were unaware of such shifts in attitud 
across the project, while an^BBFindicate4 that the opinions of SofTech 
pepsonnei had changed during the. project but the same was not true for 
TRADOC personnel. , . ^ , 

" ■ y To validate whether shifts in attitude toward SADT had occurred 

at TRADOC, two strategies, one quantitative and one qualitative, were 

O 

employed. Qualitative data were gathered (interview protocols and 



responses to discussion questions fi*om questionnaires) to determine if 
the subjective responses of specific individuals changed across evaluation 
pH||pes. 

A colonel having considerable management responsibility for 
the project believed that at "thejend of Task 2 the scope of ^he project 
was "far too global to be of atiy practical value" and favored redirecting 
the Task 3 effort to concentrate on particularly "thorny" hj^re-and-now 
problems . " 

His attCtude was expressed in the following comment made prior 

to completion of the Task 3 effort: 

I can't afford to spend a lot of my time working 

on these broad general goals. I have my own ' . 

problems. If they will work towards my particular 

problems at this time, fine, but L can't spend 

my time* and the time of my men If it's not going 

to solve my problems. 

However^,! following dissemination of the Task 3 model the 

officer's opinion was noted to have changed considerably as exemplified' by 

this later comment: 

■ — 

For some time we thought we understood different 
aspects of the^systera, but for the" first time 
we have a cleaj picture of the interrelationships 
of dif f erent^f Inctions and organizations of the 
Array as they relate to training. The raodel 
systematiz^d-^nd coalesced our knowledge. 

When questioned during the Phase 3 evaluation about wayp the 
riliVDOC' organization ha^ benefitted from the ^iroj.ect , this individual 
stressed that the work completed had provided TRADOC with ,'ia better frame- 
work within which to accomplish its mission.". 

Another colonel made these comments following th& conclusion of 

88 ■ , 



Tcisk 2: 



We went into this project to get someone to provide 
a logical layout of the training system so we could 
get a handle on it. (individuals) like myself 'haven ' t 
gotten that out of it yet. ' (Individuajn) like myself haven 
work will be of <any use at this time. .Tv j ' 3^]^ wait be- 
fore saying yes or no. 

• .■'/■; 

Six montbs later and after dissemination of the Task 3 report » 
f:'ils same individual made this comment; V \ 



•A 



Nothing new appeared in the models but they paused 
1 me to tli,iak of them (training issues) i^ 3 different 

light. The further delineation of what evaluation 
is ha^/ changed our concept of evaluation aind how it 
intei;/relates with the training system. ^ xhis year 
has sel^n the .realization • of what we've been (talking) 
about, for three years. ^ ' - _ ^ 

The 'above fcomments are typical of the way people described the 
SADT project before and following dissemination of the Task 3 model, 

Th8 second strategy employed to determine if attitudes had 
changed during the course of the project wais to compare the responses 
to 'specific quantitative questions from the Phase 1 an^ Phase 3 question- 
naire^. ^ Examinati^. o.£. these instruments (see Appendiceslfi and D) indi- 
cated that 10 questions were common to both instruments . Figure 7 pre- 
sents the intent of these questions and the mean response to each item 

- ' * . • , ^ 

^cron^ evaluation Phases 1 and 3. ■ 

Figure^^^^^Lcates that the mean responses^^to items common to 
''pi^^3 



the Phase 1 and 'Pl^B 3 questionnaires were, with one exception, higher- 
for Phase 3 respondents than for Phase 1 respondents, gi^ of tlie respon- 
dents to the Pha^e 3 questioip^aire j(n = 12) were also respondents during 
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Phase 1 (n = 13). Relatively large shifts (of appiiVxlmatv- ly one uait 
magnitude) occurred for items inquiring whether SAIjT haxi sfived time 
whether SADT was an effective design and analysis mfetl|Q(^:and whether, 
the progress, achieved had been/worth resources devot;^_d tb the proJecJ:. 
Moderate shifts (of •approximately .5) occurred for Itts inquiring about 
SADT's facility for promoting clearer understanding and focuilng atten- 
tion. Th6se results further serve" to illuminate the nature of changes 
in attitude before and after production of the Taslt, 3 model . ; 

To further, specify why attitudes towards SADT became more 
positive as the prdjectt progressed, TRADOC of f^ers were questioned in 
the -Phase 3 evaiuration questionniare (Appendix Item 22) as to why ' 
their impressions of the SADT project had <;banged*.. The general response 
given was that at the conclusion of Task 2; btily the necessary preliminary 
act ivitiej/had been completed wtiilo production of the Task 3 model was 
the goal ^of the Army's participation In this project. The following 
responses are representative: 

. More people were able to identify with and use the 
general model describt^d in the Task 3 report. 

Since Task 2 dealt wirti a specific weapons system • . 
(thd tank)', many officers were either unfamiliar, 
or uninterested in the Task 2 report. The general 
subject of 'the graining system is more widely " ' 

understood and generated a greater- degree of 
interest. Those who understood the entire piK>ject 
. kf^w that the meat of the dontract was in Task 3 
and that, Tasks 1 and 2 were merely preiiminarles . 

The Task 1 and 2 reports were not promulgated as 
solutions. They were a means to an end. The Task* 3 ^ 
report, on the otbet hand, was intended a,f or wider - / 
distribution. Thus improvement In opinion should 
■ . ^j occur. • . 



TRADOC personnel wcrq unsure of the value oJf the SADT project 
during the preliminary tasks of the project. The Task 3 Wdrk Integrated 
these previous efforts and produced a model of the training and evaluation 
system as It should be, which was the TRADOC goal for the^ project. ' The • 
complexity of the problem cremated Initial confusion among TRADOC pers6nnel. ' 
as to the goals and objectives of the pr,oject, and because the primary 
project goal was articulation of the new training system, It was not 
until the production of the Task 3 model that Individuals could see that--- 
this goal was attainable. , 

I This conclusion should not Imply that evaluation efforts at' the 

conclusion of Task 2 cdme too early- Rather, the Phase 1 evaluation 'result! 
were critical data for understanding the initial ambiguity and skepticism 
Inherent in projects such as this one which feature the novel applicatipn 
of a specific^.technlque to a highly complex system problem. 
Factor-s'^InfluencinR Interpretation of; Proj ect-wide Results 

.„ The following digcussion presents several aspects of the TRADOC 
organization which should be, considered in drawing conclusions from this 
stjdy. It is believed .that these factors 'havq* h^ suffi^iient impact on 
these results to warrant discussion. 

a/ \ Lack of command emphasis . Throughout the project TRADOC . 

officers have stated that the value and usefulness of the SADT project 

^ , ■ , " i 

was dependent on command emphasis and that the t)roject has had less.>than ■ 

.< 

full su^.port from"^ the -upper level management of TRADOC. This lack of 

* ■•• ' . ■ ^ <" ' • . , 

command em^phasis may have af fected some individual's opinions of and 

participation in the project, thus hindering the" aecomplishment of » project 

gdals.*' Although not measurable, the subtle influence th^s variable, may • 

haVe had on the conclusions drawn'can not be discounted. ^ I 



B> Ackriowledgemenif: of contrlbut lon > This factor concerns a ten- ' 
dency on the partjof some TRADOC "of fleers to discount 'the coAt^ibutions . 
of outside contractors. I Although not measured in this study, the impression 
of the evaluators ' is tl^ some officers would have little good to say about 
the work of any outside fconttact'or , regardless of performance. This 
relates to what one TRADOC officer called "the not Invented here syndrome. . 
This refers tb th6 l^eliqf that officers will tend to eihp'has.ize the. Work , 
done, in their own ishbp aiiid dismiss the work done by others, ^ ' . 

C. Influences ion the status of trainings It. was concluded, at 
the efi^ of the Phase 2 e^^aluatioit -thiat a potentially' large number of in- 
fluences may have interac-ted to produce current attittides at TRADOCj 
^ The SADT project represents only one of these potential influences. The' 
most, direct resporise f romj an officer bearing on this isst^e was from'a 
colonel who offered the following comment: . ; 

' \ . ■ : V. '/ ■ / ■ ; ■ 

;The discuss ions ,^\^ltih SofTech and examination of' 
diagrams wad onef 'of several intellectual activitl^^s 
that define^' and described where ye we-re heading in 
TRADOC. y ^ \ ' * . 

It is/ not possible Co "tease out" the amount, of influence the 
^ SofTech project has had ini bringing about the Army^s current conceptual!- 

zation of training from the contribution of other potential influences. 
•Thus, the data gathered for this evaluation must necessarily be inter- 
preted in light of the fact that this project \/as one of. s-everal activities 
. engaged in by TRADOC to achieve a better understanding oE' their training 
system. ' 
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"Sufomary and ConcluslonB V 

In Butnmary, the articulation ^of the Army training and evdi^luation T 
pystem as it should be represc^nts the most^angible direct benefit gained 
, ' by TRADOC in this project. Ther Task 3 model integrate^ the previously 

independeiitly conceptualized areas of training, system developments^ ahd 
eva^luation in an understandable and practie^al manner. Now that documen- , 
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tation of the system structure has been accomplished the actual work of 
; implementing the new conceptualization of training can begin. 

At a minimum, the Task '3 model has provided a way of communi- 
cating to agencies and Individuals both within and outside TRADOC the 
future directions plapn.ed^ f or Army training and evaluation. Prior to ' 
the productian of the Task 3 model the training^deyelopments component 
of TRADOC had not been able to fully articulate its goals andHntents. 
The Task 3 model provided this much needed documentation. 

Perhaps the -'most disappointing result of this project was that 
only a limited amount^ of technology transfer Recurred between TRADOC 
and SofT.ech. Giy^ eTiat the level of Army particip^lon in th^ project 
was less than originally anticipated, the fact that the project has had 
minimal impact on tRADOC'*s geheral analysis and planning capabilities 
is not surprising. More^ specif ically, the failure of TRADOC to provide 
additional personnel as SADT. authors limited the amount of expertise 
SofTech could pass on to. the client organization. 

In spite of- the decreased level of participation, a sm^ll 
degree of technology, transfer did occur. One TRADOC officer (a captain) 
trained in authoring SADT diagrams, became proficient in this skill. 
Additionally,- two colonels hav^ing management responsibility in the 
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training development branch^ of TRAj)OC .were exposed to the general SADT 
approach to complex system problems. .From discussions with these iridi- 
viduals during the project. It is believed that they now have a better 
.perspective from which to manage complex analysis and design problems. 

The original project goals wore that SADT would be used in 

conjunction with TRADOC to: ^ 

' ? ^ • 

(1) identify changes in Army training that 

would significantly increase cdmbat - 
effectiveness; ~ . ' 

' " • ' ■ J ■ ' ' 

(2) describe how Army ' training , testing, and 
evaluation programs will (l^p^rate after 

thq proposed -changes are Accomplished; and 

■ ■ >..''■ 

(3) plan how the changes in Army tr^Xing *- ^ 
Will be undertaken and how proj^ess will 
be' monitored and evaluated. ^ 

The conclusion drawn from all evaluation data collected from the study 

is th^t goals 1 and 2 have been accomplished to the satisfaction of 

relevant TRADOC personnel. ^ ' 

Goal 3, however, was only minimally Realized, although this 

was not due to shortcomings in the SADT technique 6r in the efforts of 

SofTcch or TRADOC. The actual implementation plan was deferred by mutual 

consent arid replaced by a more pragmatic intermediate step,; It was not ^ 

until the Task 3 model had been produced that ..the task of implementing 

the new system could be fully appreciated.- Jt was then decided that Task 4 

could better, be spent attacking more specif ic pro&iems that stood as 

obstacles to subsequent implementation o'f the full system. Given this 

redirection, the failure to develop" a plan for implementing . changes and 

monitoring progress can not be seen. as a limitation of SADT but rather 

as a reflection of the oamplexity of the context, in which thel'' proj ect occurred 



Thus, it is concluded that furthet applications of SADT such 33 
the one described -In this jp^port would be wprthyhlle. The b^sls^or this 



concli^sion is not that there Is an^ ''magic" In the SADT diagrams or br'o- 
cedures^but rathet that they, provide a mechanism .for documenting the best 
thlnkln|g^^ofc5|)rlght individuals in. the cllertt organization and Interrelating 
Important concept* in ^a clear and practical manners 

It does not. appear t|iat the project created any new , knowledge 
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about specific aspects, of the tr-^lning system but rather that j 
and communicated what was known about training In a'^coherent ^d useful 
way. This conclusion is best summarized by an officer when hesald, 
"(The project) forced TRADOC to explain the system used to outside agents. 
This is a remarkably useful exercise for smug persowS who think they 
understand wh%^* they do." ' , , ■ ; 



'; ; ' ' ■ Appen'dlx .A ' ' ' ' 1 i ' ' ' 

1 Evaluative Dimensions DeriVed .from the SofTech Proposal * 

•' y \/\ ' ■ . - ■ ■ , ' , I' ' ' ■, ^ '^ ' ■ , 

• Gpputinrilca^tlvjb Qua J 1 1 ies ^ . I 

Wliile explicit claims^ were made about the communica'tlve qualities of the. 
technique, almost all claims at -least implied communication. Communication ap- 
peared tp bd one of the unde^^'lying goalfe of the SADX process. There are three 
sufcdimehslons relevant, to evaluation of the communicative jqualities of SADT : 
(a) Does the author communicate effectively with thQ^e'ader through S>ADT dia- 
grams?;; '(b) Is communication enhanced among relevant p^^ftjtinel in the project 
by following the SADT discipline?; and (c) Doeis ' the "usefulness of the technique 
derive largely from the technique itself, ^i.e. from blueprint-like drawings, or 
from, the commWication between individuals enge^idered^^y the process of using 
the technique?" * ^' \ . i 

A". Communicative Quality of the DiaRratns . : Are SADT diagrams' readable by 
^^^y persoi^ Iprovided by Softech?'^^s 4^ message of a diagram 

clear artd unambiguoi.ts\^to the "re^der?^ Disregarding the accuracy of the diagram, 
does the Army read'^r feVl that he understands what the author is trying to say? 
'Are the-^diagrams more yoirimunicative "^than stand,ard verbal text? Do TRADOC per- 
,.sonnel believe^Key would have understood the author.'s message as easily 
through written text qr dialog? , J ^ ' 

Coimuunicative Quality of >-the SADT Dist^lpllne . " Has the SADT -diseipline 
(the iterative au.thor-comnf^nter ^cycle) facilitated communication between the 
diverse personnel participating "in the project? Is the commenter cycle more 
eff^'ctive than, written communication or conversation? -Has the SADT discipline 
facilitated coirihii^n lea tion between Army personnel abdut the ^r^lemsVof raplcily * 
c.hinging trailing procerlures? Has the discipline helped focus aj:tX\:ion ^ o 
the rel■evaTlt^1factoxs so 'that meaningful dialog. cQjald occur r^ithin TRADOC? To 



what Extent has the application of 



SADT at TRADOC led to a common conceptualiza- 



tion of the problem? Do Army pers<j)nfiel believe they are more able to c^pmiunl- 
cate among themselves about ^their tasks, goals, and ^problems as a resUlt of the 
application d£ SADT? . ^ . . j ' . ^ 

C- Impact of Improved Communication ; If communication about the problems 

' ' • ' " " ■ ■ i ■ 

af hand hae improved at TRADOC (A and B above) J has this improved communication 
helped focus attention on critical variables impacting -the effectiveness of 
Army training? Has this improved communication led to a keener undetstanding 
of the problems facing the training command . . , to a grasp of p€^ssfble solu- 
tions to those .problems? To what extent are the benefitsTof the, SADT project 
attributable to *the impact of SADT on communications within TRADQC? 

' II. Quali^ 

T^e proposal made several claims to the effect that SADT will improve 
the quality of the wo^k at TRADOC- ^/V^ these quality claims wei*e sometimes too 
vague to measure, several "claims did^cluster into three groups, that are logically 
related to-quality: atjcdracy, quality control, and efficiency' (time savings). 
These three subdimensions form the Quality dimension,. • 

A, Accuracy . Do Array /personnel consider- the SADT models to be accurate 
reprasehtations 'of the content modeled? Is , the overview model developed in 
-Task 1 a cr^edible representation of the new training progr^s to. those who are f. 
responsible \for the conceptualization and design of' ^hese new prpgrams? Similar- 
ly, is the model of the tank. weapon syst-em developed in Task 2 sufficiently 
'isomorphic with that system for Army'' users to develop conf.ijience in the model 
and to use it to deepen their understanding of the we^ipon system? _ j 

^B. Quality Control . . Bu*ilt into the , SADT discipline is ,an iterative re- 
view process that supposedly ensure^ the quality (accuracy, conslystency , and 



completeness) i^f the ongoing work. Do 'AiWy personnel perceive this metrfod of 
quality "control as eff ectlvit^^ they believe that* this process actually ensures 
the quality of the final draft of a diagram?^ If they do not see the process 
as effet^tlve, Is It because of the author's Insensl tlvl ty to comments by com- 
mep'terd • • • 'lack of Interest ,(ind/or Input, biy coramenters . • Inconsistent 
feedback ^to author ' • • • poor inf oririatl/pn graspVby experts of the content 
mcfdeled . . and/or limitations on, the, way feedback Is given? ^ 

C. Efficiency , Several claims jwere made about the reduction id. time 
resulting from appllcatlon'of SADT in i completing a structured analyslfecStTd design 

■• - . ':]'■.' ' 

problem. Do the Army's perception^ support these claims? Does the Itfiratlve 
review cycle of the SADT discipline save time? Is it a more efficient way (In 
terms of^ time) to communicate ideas? ) Does the discipline seem more time ef- 
flclent early in the modeling (A-0 and A0) ar later in the modeling, > 
more de levels? ^ ' 

At the end of^ Task 2, do Army personnel consider the amount of prog- 
ress made. on the problem as a result of SofTech's input worth tKe^lme devoted 
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to ^the project? / l)o they consider this amount of time a savings >>f deficit rela- 
tive to the t^imii it would have take'n to reach the same level of output by. more 
traditional methods? % . \ ' ' ^ 



II>I, Usefulness 

i 

This dimension, reflecting tj^e largest single cluster, represents claims 
about the utility or usefulness oi the technique. Taken together , these claims 
expli(^tiy state trh^t SADT is mor^a than a decomposition technology or engineer- 
ing drawing systeni', i.e., th^ it will facilitate the planning, implementation, 
management, and evaluation capabilities of TRADOC personnel by providing con- 
ceptual insights into the problems being modeled. ' 



The -evaluation of this dimensibn will be conducted at two! different' stages ; 
(a) an evaluatio,n of SADT's useftilness in Tasks 1 and 2 (inj^orp.brated in this 
evaluation report) /and (b) given the Army pei^'sonriel ' s experience with SADT in 
Tasks 1 and 2, their perception of the usefulness of SADT f hen' applied' iri 
. Tasks 3 and 4 (not sch^^ed fol^ completion prior to 'the^reparatiort of this 
repc|rt) » - * 

^' Ugefulnes s in Tasks 1 and 2 , 6id the models produced in Task 1 lead 
to a clearer understanding V^f the problems being modeledf Will these diagrams 
be helpful' in planning changea in Army training? Did model define what' 

types of change^^t'e^being planned? ' ' 

j:n Task 2, did the training" innovations havl^ng 'the greatest impact on ^ 
combat readiness and battlefield ef feet iveness .become sufficiently Visible" as 
claimed? ^ By the end of Task Z/'^did SADT contribute to' the/i^glat ion of inno- 
vations Ahic^ to Army training? /. 

B» / Usefulness in Subsequent Tasks . Do Army personnel believfe- SADT will 

be ef'fecti'^ in elaboratitig the Task 1 model? Given their experieVeVith SADT 

in Tasks 1 and 2, do thfcy b^lievc^the more thorough model produced in Task 3 

and the innovation plan (Task 4) will increase TRADOC's analysis, planning, and 

management capabilities? In addition to their utility for the present evaiua- 

tlon effort questions such as these ultimately will be. used to* determine " • 

whether the consensus of pinion at TRADOC supports the continuation of the 

project and/or the use of SADT in expanded contexts, . . '^J^ 

' Spe cificity ' ^ 

• * 

This dimension reflects the Army's perception of':SADT's ability to produce 
models ^tliat meet a spedific need, such ^as describing what functions a system 
must perform, specifying how a system should be designed, or how a system should 
be managed or maintained. Answers to these types of questions provid^ some 
initial information bearing on a decision about-expanded usage of SADT in 
' ■ 102 ' . 
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the Dob environment. 

•A second area af , Specificity Is whejt:hei;^'A!t'my p.eVsoqnel can obtain specific 
-information from the modolfl.' If an Army officer Aeeds to know something 
, specific about a system, can he look at the SADT model of tl)at system and findf 
what^he, needs? Could, for example, a microfiche ^ADT 'model fjll:e serve as' a 
valuable reference for Army training perso^nnel? • ^ • • ^ 

. These evaluatlye dimensions and their respective claims a:]fe Summarized ^ 
in the following table. • ' " • 
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Poc/or Structure of Claims by Evaluatl/vo Dltnfinailonfl 




II , r III ' IV 

Communicative QuoUtlea Q^flHty \ j Usefulnesb Specificity 

Dlapma Discipline , kpact 'ta^ J"^^^^; Efficiency Subueqient ' 



CUlma abdiit SADT l^' ' , , < n.' ^ ni. Ill 



Control _ ^ 1 and 2 Taakfl 

IV 



1. Identify chibngea Increasing 
^ combat effectlvaneas 



2.' Accurately deacrlbe hlbw inay . » ^ 

H ^trilnlng wlli operate . ' '^v 

0. ' ._, . ' :■ ' ■ 

^ 3. / Help plan how.clMngea will' 
beaonltered . 

, 4. 'Plan hotrf prog reaa will be ^ ' 

monltirid 

5. Improve THADOC'a analytic' 



\ 



afid jlannlng ability - v ^ \ \i I 



6. (Plan how training changes 
should be accomplished' 



7. Unde^rstand cj)inplex' ■ > . ^ , /''r' ' '| • ../ 

lntirrelationfihl|i8 . , X ^ 

■ ■ ' • ' ' •' ■ ' .. . 

. 8. Focua attention pn , . ' . • ' . ' 

Innovationa with greatest . ' ' » ■ < . ^ 

potential for; incr^aaing ^ , ' 

combat effectiveneaa \ ' " / 1 , ' - v X ' ' ' . 

9. ,Plan and Bchfidule how • , - , , 
. ' inndvatlona will be , ... ' ' • • ; " . " ; 

tpcorporated ; ' ' • ' • / ' It 

10, Plan how ef feet ivenesfl^ ^ . * . ' . , ' , , 

Ijy'^ of Innovationa will be' . ' 
J ' evaluated . ,1 ^ ' ' ' ' ' X 
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11, Ui^dersUnd Mjor changes ' * , ; ^ 

In Array training ' - X ^ ' ' ' I ^ ^ 

^ . 12, Plan nilijor changes In ^ v' ' ^ 

' Army tralrflng ' X 

13, Describe new Amy training . ^ ^ * 

and UstlnR prograina ■ V ■ . X 

' ' • '■ 

U.- Derive training / 
Q ■ requirements ■ ^ ' ' i It 

. Ui , ' ' ' ' 

15. Understand aspects. of training ^ 

that icipact combat effectlveneiifi ^ X 

, 16. Evaluate flpproachea to meeting 

, training requirements K 

17, Improve quality of resulting . r 

wort . , ^ . ; • .XX X ■ 

, 18, Identify changes that increase 

effectlveneaa of ^weapons ayatema " . X 

19, Show how changes are related to ^ ■ 

existing training methoils ' , X 

h ' ■ ■ ^ ' ' ' 

' 20. Improve quality of work on the \ 

5 taakB . ' \ ' X , x X 

21. CownunlcateB results among ■ \ ^ 

' » analyata, deslgnerfi, users, , 

and managera , . X • , X 



22. Integrated approach to 
managing large complex 



EKLC 



23i' Coiimunlcates. results among people , - ^ 

with different interestfl and back' ' ' ^ 

■jiycjindB ■ . X' X ' / 
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" C 



III 



A I 



III 



2i« DowntB logical relatlonehlp 
of components to total aystem * X 

25, Consistent decompoalMon of a 

. ayBteiB". . ' , , 

26. Effective QoiMunlcatl9n, between 
originator and usen of 

' descriptive modek X 

\i ■ ' , . 

J 2?, Will yield unusunUy clear 

0 deficrlptlon of problem X 

0 

29. CoiDplete decompoBltlon of 
0 system I 



30* Highly readable decooipoaltlon 
of a system 

31.. HlghllghtB interfaces', betveen 
coiiiponcntQ 

32* Defines changes being planned 

33. HUl'descrlbe training aysteoi 
operation after changes are 
Incorporated 

34. Hill be effective In training 
comnd 

3J. Hill have utility In training 
cownd 

' ]L Describes Interrelatloiiahlps 
between innovations ^ 



37. Highlights complex 
|j ' Interactions 



V 



( \ 
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38. Integrated appi;oach to 
planning, analysis and '4.^^^ 
design . 'iV 

39. Ensures quality and 
configuration control ■ 

' 40. Ensures consistency and 
completeness 

^ 41. Interviewing atandardlzes 
^ means of capturing Infor- 
mation ' 

42. Interviewing organizes 
process of gathefing fact^ 

43. Defines requirements , 



'I 



II. 



.7: 



11. 



Ill, 



III 



IV., 



' 44. Describes what functions 

0 

a system must perform 



45. Specifies liov a system's 
use or construction la to 



46. Explains a sygtera'a use 
or inalnftenance 



ft 



47. Specifies a systei's 
design 

48. Allows a complex system 
to be understood 

49. Critical Danagement and 
, technical review built 
Into overall process 
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50. Constant attention' given 
♦ to. quality of work In , 
progressl 

5L SlnpUflel^ testing of - 
\ ^ ' all Interfaces for <" 
conslatency and completeneBs 



. t 



$2. Allows anaiydls and deBcrlp- 
tlon from aevi^ral viewpoint 

^ Q 53, Hill reduce (elapeed time- to 
•no complete 5 t^aska ' / 

54, UIH save blme'.oij thlp 
project * • - / 0 



^5, Id expected to reduce 
elapsed time to compete \ 
5 tasks : ' • 

^56, la particularly powerful ' 
■for planning . s. 

57* Involves mo/e th^ln a 
decomposition technology ' 



^8, la \ 9y|teiiiatic methodology 
fpr pexfomlng planning 



/llOlE: Claims 8, 11, 13, 28, 31, )8, :«i,57, and 58 are equally aJlcable to two (ilmenslois. 
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Appendix B , 
Phase L Evaluation Questionnaire 



Current Position 



Months in Current Position 



Ttytal length of experience with Army training 
^ and "testing programs 



Have you taken the Sof Tech Reader Course? yes / / .no 

Have you taken the SofTech ' Coriimenter Course?' yes / / np / / 



1. To what extent do you feel you were adequately trained to read; 
^ SADT diagrams? . ' 

1. not trained at nil , 

2. poorly trained - ^' 

3. trained to a working knowledge 

^i.j. fairly we].l trained 

5. ex'per tly • tra Lned 



1 1 1 

117 



To what extent do you feel you were adequately trained to cotmnent 
on SADt diagrams? ^, - \ ^ ^ 

1 « - n o t . trained at all , » " ' > 

. • " ' . . • '.'l-^ 

2*_^ poorly trained 

3. trained to a working knowledge 

4 . fairly well trained ' 

5^ — — ■ — *— - ■ , _ 

5. _ ^expertly trained' 1 - - 



In general, to what e^^tent were the SADT diagrams clear and 
unambiguous? (Conipiote e^ch column separately ^ \ 



first revised final 

drafts . versions vef anions 



1 . Diagrams were always 
ambiguous 

2. ' Diagrams were mostly 

ambiguous 



3. Diagtams were occasion- 
ally cleaj ' / V ^ V . ./ j:'.. L 

4< Diagrams were niostL^ clear ,^ 



5. Diagrams were always clear 



To what extent do^you think it is necessary that written ^ text; 
containing a more complete explanation of the content modeled 
accompany SADT diagrams contained in reader kits? 

not necessafy ' ' . • 

jTiLght be slightly helpful 

probably of some help 

A. ^ ^would he fair^; helpful 

'5.^ would be very helpful . 




I IZ 



- ■ ■ ■ ■ ■ , 

In your opinion^ do SADT reader kits allow adequate commifhicla tion 
between author and commenter or should discussion between author ^ 
and reader be- a required part of the SADT process? 

1 . SADT diagrams are an adequate means 'of conununication ; 
* no discussion is necessary 

2 discussion between author and reader at the request of 

either party (the, present procedure) is adequate 

5-. ^inclusion of required discussion between author and 

^reader would be helpful 

4. inclusion of required discussion between author and 

reader is" essential " " / 

~ ■ ■ . .■ ■ - / ' ■ 

^ > ^ .... • ' - - ■ / 

To what degree, do you believe SADT diagram^ -are an ef.f ^c tiA^e^ ^ay 
p*f communicating to trained readers the .substance' of l^troviding 
Army training, -testj;ng, and evaluation 'programs? " . *: 

1. ^hot effective ^ 

2^ - rare ly^f feet I've " . ■ 

3. sometimes effective"" - ■ . T . '^: '^''^ . ^ ■ , 

4. f ai riy . e f f ect i ve \ 

5. very effective 



To your* knowloclgq, to what extent have SADT diagrams generated- ; ^ 
either formal or informal communication about Army training 
proceehires among TRADOC personnel dire c tly Involved with the SADT 
project? . 

1. none at all i , . . - ■ - ' 

2 .:-;;;_aMittle ,• . • ' ^. ^ : ' ■ ' 

1. some ^ ' • 

6 . i a i r amouu t ' .. . , . ■ > ■ 

S. . very sig.nilieaiU anu)unt ' . 
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8. To your knowledge, to what extent have SADT diagrams generated 
either formal or informal Cbmmunicatlon about Army training 
procedures among TRADOC personnel no t directly involv.ed with the * 
SADT project? [\ . 

1 . n one at all 

2 . a little 

3 . -some . * . 

fair t'lmount ^ , ■ . ' — 

5. very .sLgnitMcant aniuunt 

■ f , ; ' . 

9. To wha/t extent have SADT diagrams helped focus your attention on 
training^aiul tes ting,. varlabl>,^s whiich you had not considered 
prevlous'ly ? . ; . ; 

1 not at all ' 

• ' 2 . _ Very li 1 1 Ic : / 

3 somowha t * . ' - 

4. to a fair extent 

5 . very much ' 

10. Lo what extent do you r)cMeve the henefits of the' SAP ' project at 

TRADOC,. if any, are due to ,.tK?, diagrams or models tb.-nseives, i.e., 
the tangihie products ' or drcawiog^? J as opposed to tht: process of 
communi catioit between Tl^ADOC taf f mejnbets which' nua liave been 
. S t i in ivl a t e d by the nu> i M n g p roc e s s ? 1' f you feel no benefits " • 

accrued at all relevant to this question, check o .y the following: 

' ' ;* / / wo bene fits at al T ' . . 

■ ■ ■ ■ > " 

1» luMVet lts iMUirely product-diagram relat(ul 

.2. lna^efit^^ mo:U 1 y product-diagram volattnl ■■. 

"■ } 3; bono r i t s split abtuiL evenly between procUicts produced \ 
cUid ]>Koi:r\'ises s t i-imi 1 a t ml 

. . bi'UiMit!; niostlv proct^:^; rola.t'cnl 

^ • ; bong t i t ' iai t i ri' 1 y pr^n^oi;.'; I'olated 

■ • ' ■ \ ' 
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11, to what exten^ do the revised diagrams that you h»ye-«een of the/' * 
.new- Army training and testing programs accurately represent the 
content modeled? . ' ' ^ 

/ / not enough Information to judge ^\ * 

1--; ^not accurate , r - 

- 2. trough approximations'-' • 



3. __varia^|le in accuracy, depending on content modeled 

4 . ■ > fairly accurate / ' 

■ 5._ ^GXtremoly^accuratc * 

12'. To what extent do the /evLsed diagrdims that you have seen of tlie 
J^^OJS weapon system accurately represent that system? V 

. ' '• l_/ not enough in.fonnation to judge 

1.^ not ^accuratq^. 

! 2, __rough approximations 

' 3. ^variable in accuracy, depending on content modeled 

4. ^^^^il'^ly accurate 

. oKL reniely acem'aLe . ^ 

13. Is the .iutho.i*-L':6mmeiiter review cycle usexl'by SdfTcch in the 

. prixiuc t ion of S.ADT diagr.'imr. an effective nieaas ofc tonsuring the 
(lualitv (^*r SADT mcule I s^^^ ' " . " 

1.. ._Jio.t'at all i>ti(H'llvc 

* 2. not very iM ftu't ivr. 

r,onic\^/h'at tM' tec t i vc ' _ 

4 . t a i r" i Y c t ice. t ,1 vi^ ' ' ' 
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How confident: are you thar; method of ciuali£y control guarantees 

that the final dlagram^y^^ aat^ufateVas .possibly? 



rvot at- al). confident 



.21 very' little confidence > * " ' %' 
3> • some confidence* , ^ 

^. fairly^ confident ^ ^ 

5 __very confident :v . ■ 



What degree of confideli.ee dd :you have thcit your writteri .doiimients 
on SADT diagrams were actually taken into account in, producing 
final draft models?^.' ; • ' ' ' 



1. no coul id"ence at all ' 

2. ^low confidence ■ 

^ . s o me c o n f i d tui c ' - A \ ■ 

^+ . fairly high degree of. cgnfidenco 

3. very i^ilgh degree of confidence 

til your o.pinitnu what extent: has communicating to SofXech 
authors,.by way oV. the autlior-commenter cycle, ratzlier than by 
discn..ssipn and written text , rp^-SAil'to'd in a time savings? 

^ *„: no , ;t: 1m^.^•^ was saved ' . - . , 

- \ '<} l i t: tit* saved 

K some I: iiut^ was- saved 

. a t a Lr anunnU oT t ime wa.s .savtnl 

■ ' ' ' * / 

. , a \;reat (Uvil of time was. saved 



17. To what extent has the application of SADT at TRADOC saved time ' 
relative to other approaches (e*g., In-house TRADOC committees, 
outside consultants) that might have been employed to communicate 
Army training procedures? 4 

• .' . 

1. ^has not saved time - 

2. Una saved very little time 

3. h as saved some time 

4. ^has saved a fair amount of time 

5. has saved considerable time 

^ ' - ' ■ 

18. How cost-efficient do you believe SADT is comparedTvto other 
techniques having the same general purpose wit^ which you are 
familiar? 

, / / unfamiliar with :other techniques ' 

1. a\l others are superior to SADT ^ 

2. _many others are superior to SADT., 

3. ^^ ^others and SADT are about the same 

4. _ _a few others are "Superior to SADT ^ , 

5. SADT is superior td all others r 

19. In your opin&n, approximately what percentage of actual time could 
/ be saved, if any, by using/SAIJ^T on a design and analysis tasl^ 

instead of traditional prqpediires? ^ 



1 



117 



To what extent did the Task' 1 model of Army training provide you 
with a clearer understanding of the Army's training, testing, and 
evaluation progr<ims? 

/_/ unfamiliar with Task 1 model . j 

1'^ less understanding than before project 

2 same understanding as before project 

3-_j a slightly better understanding 

a noticeably better understanding 

5 » a much better understanding < * 

Do you believe the SADT models produced by SofTefch thus far will 
actually be use^ in planning needed changes in Army training? ^ 

1. _ will not be used . • / 

2. ^wlll'be used to a small extent 

3. _-_ will be used some, depending on content modeled 

4. __^wlll be used to a fair -Ktent 
5> w ill be used^ extensively 

To what extent do you believe the SADT diagrams produced thus far 
will actually help* in deriving training requiremients? 

1> \ will not help 

2> w ill help to a small extent 

3._ will help some 

A, w ill help noticeably 

5n ^will help considerably 



L 
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23. In your opinion^ can SADT effectively used to^elaborate upon ^ 
already completed diagrams and to accurately articulate an extended 
final tnodel of Army tralning^and testing during the next phase of 
t^e project (Task 3)1 i 



definitely^ not 



jprobably not 



3, unc'ertain 

■) 4 . ^probably yes 

5. definitely yes 



24. To what extent will SADT models actually increase TRAEfaC's analysis, 
planning a>id management capabilities? ('Complete each colupin 
separately.) \ 



little 

3 . some 

4. a. /air amount 

5. very much 



Analysis . Planning \ Management 



25. 



Do you believe that jSADT can help identify training requirements 
that have a significant effect on combat .'readiness and battlefield 
effectiveness? 



1.. 
2. 
3. 
.4.' 
5. 



definitely not 
a little 
somewhat 

to a fair extent ' 

to a significant .extent 



1:? 
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In your opinion, would a compleite SADT model of Army training sefve 

a valuable f utdre reference for Army personnel' responsible .fo^' 
providTng training? . 

1. ■ w ould not be of any value _ , ' * 



2. wc^'uW be of little value 

3. w^uld be of some value 

4 - - would *be fairly- val 



5. 



ly- vali^ble 
would be very valuably 



27. How Practical do you believe it is for TRADQC personnel to use SADT 
for ^ost design and analysis' probjLems facing the Training and 1 
Doct/rine Command? 

_^very impractical , ' , 

fairly impractical . > 

^marginally practical 

fairly practical 
— i 

_very practical ' . 



How confident ware you, in the comments you made on SADT diagrams? 



2. 
3. 

5. 



not at all confident 
^barely. confident 



somewhat confident 
fairly confident 



very confident 
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29. Of the dlagirams you read, appro»imatijBly what percentage were In 

content areas in which you felt ypu /did not have sufficient eWeri- 
. ence to make meaningful^ comments? if you would like, to be morW 
precise in your estimate, do so in the space provided: 

■- . ^ . 

1. 0-20% ; '^^ ^'t ■ 

. .2. 21-40% ^ , ' . 

3. 41-60% ; y , ■" 

'A._ 61-80^ _ : , ■ ■ ' \ , 

5. 81-100% ' ' 



30. About h(|w often did you consult other'TRADOC personnel befo°re 
comment ing„^n an SADT diagram? If you would -l^ke to be more 
precise in ^y our estimate, do so in tne space provided: . %, 

1. 0-20% 

2. 21-40% 

. 3.J _4i^-60% ' / „ ^ 

' 4. 61-80% 

5.^^ 81-100% 

■ ■ / . . > ■ 

31. At this time, how/critical do you believe it is, to implement; 
changes in Army t/iraining? ' 

1 . no need presen?^y exists 

1. ^needs to be done but not immediately (low priority) 

3. somewhat critical (medium priority) 

4. fairly critical (high priority) 

5. ^very critical (urgent priority) ' 
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32. * To what extent do you believe SADT models will pro'vlde the Impetus^ 
for ^chah^es In Army training and testing programs? . . * 

.1. v l/ll not provide any impetus for change 

■ • 2. . will contribute to change to a smdll extent 

3, w ill provide some impetus for "change 

"4.^ 

5. 



will contribute to change to a fairly large extent 



_will provide the tnajor impet.u^ for change 



33. To what extent will SADT be useful in iclentifying existing 
organizational inef f iciencies 'in the current .training and 
testing procedures? . ^ 



2,/ 



3, 



no use 



not much use 
some use 

_a fair ambunt of use ^' 
a great use 



34./ In your opinion, 'will the api>lir -'f ' n or SADT lead to conceptual 
insights about alternative methods of conducting Army training r 
; and testing programs? ■ 

/ •■ . . \ ■ ' ' ' 

1 . ^np t at all 

I. • ' ■ 

2, __ ^very little 

3- some ; ' . 

4. ^to a fair extent' ■ 

5. ^to a great extent 
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35. In your opinion, will the use of SADT ultimately affect the 
training cortcepta currently '^held by TRADOC? * 

1. not at all 



2. _^very little. 

3. to some exterii; 

/ ^ . 

. , \^ ^ . 

4. to *a considerable exten.t | " 

5. to a great extei>t 

" : - > ■ / . ■ 

36. In your opinion, would it be useful to TRADOC*s mission to 
train some Army personnel to be SADT authors? 

1. ^no use ' 

2. _ _slightly liseful 

3. ^of sdLe use 

4. fairly Useful 

- 5. jvery useful j / 

^ " ' ' i 

37. .In your opinion, has the amount of progress made by this project 

on the problems facing TRAtXDC been worth the » amount of time Army 
personnel have devoted to it? 

1. _^ definitely has not been worth the time ' 

2. generally has not been worth the time 

3. _ ^h as .varied with the content -being modeled 

4. generally has been worth the time 

5. all time spent has been worthwhile 
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38. To what extent did 'the Task 2 model of the -tank weapon system 

provide you with a clearer understanding 6£ the Army's training, 
testing, and evaluation programs? / 

I I unfamiliar with Task 2 model » 

■ ' r 6 

• •» . 

1»_ less understanding than before project 

2. ^ ^same understanding as before prc^ject 

■ ■ ■» . . 

3. _^a slightly betfeu* understanding . - ^ 

a noticeably better understanding 

5._ ^a much better understanding^ ' * * 



Please respond to the ^following questions by writing your answers 
In the ^pace provided. If "you need more space, continue -^n the extra 
pages at the end of this booklet 6r attach an -additional sheet . While 
some of the questions could be responded to~wrth a brief comment, I 
would like to a^k that you elaborate on. your responses whenever 
possible . • ■'^ 
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From your Gxporlencc In revtaing SADT diagrams, can you recall if 
there were general content areas in, which diagrams were particu- 
larly accurate and other areas wher^ they were particularly 
inaccurate? What variables, in your opinion, might account for 
differences in the accuracy of SADT diagrams? ^ 
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AO. Could you Identify any specific bene fits which -may have ^accrued 
to the Army as a result of its participation in tHis, project 
thus far? 
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41. Drawing on your cxperlpnt:c with SADT, what are a ^ew of. J ts .inajor 
y^t'rengths arid weaknesses? " . 




hte there other personnel at TRADOC who were h4(P:nclikled : in the 
SADJ project who may have had more experience or closer contact 
^ith the content being modeled than yourself? If so,, why^ dp you 
believe they were not participants In the project? 



I 
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To your knowledge, have tljbre bo^vn any obstacles wlitch may have 
prevented SofTech ,from dotijg the best job^posajlble at TRADOC?*. 



I 



1: 
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♦ 4. I)o you have other thou^^hts or commenLs about SADT which you 

Miave not had .the Opportunity to express? 




> 
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Appendix C 



Phase 2 Struct:uro(l Interview Form 

1. (7)* To your knowledge, to what extent has the SofTech project fac- 

ilitated either formal or informal co'mmunication about Army 
training procedures among TRADOC personnel djlrectlj^ involved 
with the SADT (^iagrams (comment ii\g or authoring)'? 

1, none at all 

2. _ a little - . 

* 3 , some , 

4, fair amount ^ ' . 

5, very signif icanj;^, amount 

2. (8) To your knowledge, to what extent has the SofTech project fac- 

ilitated c^lthor fotifn'al or informal communication about Army 
training procodurtVa iimong TRADOC personnel not directly Involved 
with the SADT diagcitms (coimnenting or authoring)? 

1/ none at all )^ 

2 . a li t^t 1 o . . ' V 

4. fair amount 

verv si f icant amount. 



\. (0) To what oxto'nt lias t ht*. .^olT^nMi proiect holpod t oeu.s your <U t 
\ iow o\\ training ami tost iug variables whith you had not con 
s i do rod provimisly? 



en- 



1, not at ctll 

V o r V little* 
somowhat 

4 . t o a Lii r <'Kt oat 



^()m-;tIon niimbof mi t lu IMiaso 1 iino.s t i cmum i ro 

Jl . > I 



4. (11) To what extefit do the revised diagramss that you have seen of 

the new Army trainin g and t£sHjig^ £rograms accurately represent 
what thjC Army training and evaluation ^system should be? 

/ / not enough information to judge 

/I. _ not accurater 

2. rough approximations 

3. variable in accuracy, depending on content modeled 
'4. fairly accurate 

5. I extremely accurate 

5. (20) To what extent will the Task 3 model lead to a clearer under- 

standing of Army training, testing, and evaluation programs? 

1. less understanding than before project 

2. same imderstandiit^ as before project 

3. a slightly better understanding 

4. a noticeably better understanding 

5. a much bett(>r understanding . 

b. (21) Do yi)U belii^ve the SADT models produced by SofTech thus far will 
actually he used in r>lanning needed change^' in Army training? 





will 


not be 


used 






, wa|. 

i 


Uik^d 


t(V a small extent 




3. 


w i 1 j 


be used 


.some, dep^-'^i^^i^iJ. <^^n 


content modeled 




will 


be uked 


1 1> a 1 air extent 




S . 


will 


l>e usiHl 


ext ens 1 ve 1 y 


, 0> 



1 \> VJ 



ERIC 



/ 



7. (22) To what extent do you believe the SADT diagrams produced thus 

far will actually help to e^tahll^h training requirements? 

,1. vlll not help 

2. will help to a small extent • ' ^ 

3 . will help some , , 

" ' • ■ ■ ■ I... . . 1 

A. will help noticeably 

,5,^ will help considerably ' ; ' . 

8. (23) Have the Task 3 modet^^ effectively elaborated those developed 

' in Tasks I and 2? ^ . 

■ r- ■ ' ■ * ^ 

1 . def iV^rtely- not 

2. probably not ' • ' . 

^ V , nncer ta in - 

A. probably yes , . : i. 

5., tlePinitely 

9. .-(2A) To what extent will SADT models actually increase TRADOC * s 

analysis, planning, and management capabilities? (Complete 
eaeh colvimn se[^ar^ltoly . ) ' 

Planning Analy5;i.s ' Managoavent 

I . none 

2. a 1 itllo . ' : 

3 . some • ' 
4 . • .'I f a i r .iiuount 
S . viT V much 



\ 



V 
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10, (26) r *Tti' 5four opinion, would a complete SADT model of Army training 
' serve as a valuable future reference for Army personnel respon- 

.. ^ sibie for providing training? 



1. 


woulid 


not 


be of any. value 


2. 


would 


be 


of little value 


3. 


would 


be 


of some value 


4 ; 


would 


bo 


fairly valuable 


5. 


^ would 


bo 


vf^ry valuable 



11, 
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(27) How. practical do you ' believe it is for TRADOC personnel to use 
SADT for most design and analysis problems facing the Training 
and Doctrine Command?. 

- ■ - / , . 

li very imirractical 

■ ' ■ ^ ' *" 

2. ' fairly impractical 

"3. niarginally pr^vctical /. 

^ ^ 4. fairly practical 

3. _ very practical - > * 

To what extent to yoii bel leve SADT models will be useful in 
hrin<;in^; about I'hanp.cs in Army train lnj:\ and test: in f> programs? 





- (Not 


i nc IulU 


i! i n or ip, ina 1 ([uest ionna i re . ) 






1.'. 


w i 1 1 


not provide any impetus 1 


or change 


\ 






wl 1 1 


rent r ibute to change to i 


I smal 1 extent 








will 


provide some iiiipt^tius\ for. 


c-hanj',(^ 






A. ;■; 


w n 1 


e(*iit r i but e to chanp,e to i 


1 ftiirly larp^e 


extent 




3. 


will 


proviik^ .t hc^ ma jor . im'ptU uj 


; for clianj'.e 
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13. (33) Has SADT beep' useful in identifying existing organizational 

Inefficiencies, in the current training and testing .procedures? 

- ' r 

1. no use 

2. not much use . - ■ ^ 

3 . ^ some use ' ' , / ' . ° 

4. a fair amount^ of use . 

5. a great use ' / «^ 

14. (34) Has the application of SADT lead to conceptual insights about 

^ alternative metliods of- conducting Army training and evaluation 
programs? 



I. 


not at ' all 






2. 


very little 






3. 


some 






4. 


to a fa i r extent 






5. 


to a great ^extent 




V 



15. (35) Has tUe use of SADT had an impact on the training concepts 
currently held by'TRADOC? 

K not at all * • . 

1 ' ' ' 

2 . ^ V t.^ r y I L 1 1 ' , . ■ ' 
3. to some extent 

. , 4. ^ ^ t 1 c'onru d(^t;ahle extent 



i-ii. 



/ 
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16, (36) In your opinion, would it be useful to TRADOC's mission to 
■ ' train sbi^Ci^ Army ^ personnel to be SADT authors? 



i. 


no use ^ ■ ' 


2. _ 


slightly useful\. 


3. _ 


of some use 


4. _ 


fairly useful 


5. _ 


very useful 



17. (37) In your opinion, has the amount of progress made by this pro 
jec.t on the problems facing TRADOC been worth the resources 
the Army has devoted to it? ' 

1. definitely has not been worth the time 

generally has not been worth the time 
has varied with the content being modeled. 
generally has been worth the time 
all time spent has been worthwhile , c 





2. 



3. 

4. 
5. 



Al4)endix D 
Phase 3 Evaluation Questionnaire 



Check the alternative most .congruent with your sentiments to each question 
Feel free to elaborate on your responses in the additional 'space provided.- 



1. To what degree do you believe SADT diagrams are an effective way 
of communicating to trained readers the substance of providing 
, Array training, testing, and evaluation programs? 

1. not effective 

_ — . — ^ ^ _ , 

2. ^ '_ ^arely effective 

3. ^ sometimes effective 

4. ■ fairly effective 

5. ' very effective 



To yoitr ^knowledge, to what, extent did the SADT project generate 
either formal or Informal communication about Army training pro- 
cedures among TRADOC personnel? 

1- none at all ^ - 

2. ' a little 

3. somalj^, 

'^iv.^ fair amount 

^" " very significant amount 

To what extent did SADT diagrams help focus attention on training 
and tei^tlng variables which had' not been considered previously? 

K not at all 

1 . I 

2 . very 1 1 t't le 

3. somewhat^ 

4. * to fi fair extent 

5. very much ' ' ■ ■ 
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To .what extent did SADT model's - elucidate the interrelationships 
between the various components of training? 



1. 
2. 
3. 
4. 




To what extent wete the benefits of the SAI^T project at TRADOC, if 
any, due to the diagrams or models themselves, i.e., the tangible 
products or drawings, as opposed to the process of cOTBlunlcation 
between ODCST staff members which may have been stimulated by the 
modeling process? If you feel no benefits accrued at all relevant 
to this question, check only the following: 

no benefits at all 



1. benefits were entirely product-diagram related 

2. benefits were mostly product-diagram ^related 

3. benefits were split about evenly between products 

- ' produced and processes stimulated 

4. _^ benefits were mostly process related 

^' ______ benefits were entirely process related 

6. To what extent has the application of SADT at TRADqC, saved time 
relative to other approaches (e.g., in-house TRADOC -committees, ' 
outside consultants')- that might have been employed to develop ^ 
integrated system cohcept of training? ^ 

1. ' has not . saved time ^ . * 

2. has saved very little time 

. 3/. has saved some time - • 

4. has 'saved a fair amount of time 

5. has saved considerable time 



..../■' 
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To what .extent -did ^ the Task 3 model of Army training provide a 
clearer understanding of the Army ' s *»t raining, testings and evalu- 
ation Programs? ^ 

; unfamiliar with Task 3 model 

_ less understanding than before project 

2- ^ same understanciing as before project . ' 

^' a sligl^tly better understanding 

, ^ ' ' . ' ■ 

^' ^ ' a noticeably better understanding 

^' ^ a much better understanding 

■ .. ' . ' / 

Do you believe the SADT models produced by the SofTech project will 
actually ^e used in planning needed changes in^A^my j-^-gining? 
1 . ^ ' \ N 

i- ^ ^ will not be used 

^- 4^ will be used to a small extent > : " 

3- _1 will be used so 



me 



^- _ will be used to a fair extent 

" will be used extensively - 

Has SADt helped identify changes in training procedures that would 
have a significant offset on combat readiness and battlefield 
effectiveness if instituted? 

definitely not 

^' _ ■ a little . 

somewhat ^ 



— to n fair extent 
5- ^o a slgniricant extent 
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10. In your opinion, would the complete SADT model of training serve ^ — . , 
^ as a valuable future reference for Army person^iel responsible fo/^\^\. 

providing training? ■ v , . Y 

1. would not be of any^yalue "[^J ' 

2. • Would be of litt/^ v)llue 

3. Would be of some Value. 

^' _ would be fairly valuable 

5- woulc} be very valuable 

11. In your opinion, is SADT an-ef fective approach to the design and 
analysis problems encountered by an agency such as TRADOC? 

1. ^ not effective ' \ 

2. ' • - rarely effective . 

3. marginally effective 

A. fairly effective 

5. very effective ; 

12. To what extent do you believe SADT models will provide the impetus 
^ for changes in Army training and testing programs? 

I.. will nut provide any impetus for c^hange 

2. will contribute to change to a 6^all extent 



3. will provide some impetus for change 

A. ^ will contribute, to change to a fairly, large extent 

5. will provide the major impetus for change 
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13. To what extept were the SADT models useful in identifying existing 
organizatiQiial inefficiencies in the current training and testing 
procedures? \ 

1. no use ' ' 

; . 2. " not much use 

.. .3. some use . ... 

' ' ' ..... 

4. ' a. "fair amount of use 

5. ■ ^ a grba't use ^ > 

^ ..... '-' 

14. Has the application, af SADT led; .to c^Snceptual insights, about , 
alternative methods" of conductl*ng-: Army training and testing v 
programs? 

1. ^ not at all < . . 

■ 2. _ very little . ' , ' 

'3. - - some ' ■ . 

' 4. ^ to a fair extent 

J 5. . to a grcc^t extent ' : ' 



15. In your opinion, will the SofTech project ultimately affect the 
training concepts hold by TRADOC? 

1. not at all - 

/ • . 

2. very Little 

3. . , to yome ox Lent 

. • • ' . - i . . _ to a considerable extent 
5. _^ to a p,roat extent 



1 -i •;• 
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15. In your' opinion, would it be luseful to TRADOC^ mission to train 
adjditional personnel to be SADT authors? , , 



1. no use 

2. . Blightly useful 

3. " " , . '•' of some use 

4. ■ . fairly useful 

5. ' very useful 



17. Beyond the specific models produce<;ly to what extent has the 

experience of participating in this project increased TRAD6C ' s 
general analytic and planning capabilities? 

1. , the experience has had no effect on these abilities 

2. _________ the experience has had a minimal effect on these 

abilities ' /; • ; 

3. " the experience has had an effect only in the . « 

particular content areas modeled *^ 

' 4. • - the experience has had a noticeable effect qn the'be 

abilities / 

' : . ■■■■■/ 

5. the experience has greatly increased these abilities 

' ~ within TRADOC 



18.' In your opinion, is it realistic to expect that TRADOC ' s general 
analysis and planning capabilities would increase as a result of 
their part icip^ltion in a project such as this one? 

1. _ not i\ realistic oitpectation 

2. .. - definitely a realistic expectation 

\ 3. . insuCCicient information to judge 



/ 1 ^ 



. r-^ ■ - ■■ ■ , " ... 

=■ ■' , V ^ " .■ ' ■ 

19/ In your opinion, has the amounb pf^^rogress made by this pVoject 
on the problems facing TRADOC been worth the time Army personnel 
have^ devoted to it? • . 

1. _^^ Kie finitely has not b.een worth thd time. • ' 

^' 2, , , generally has not been worth the time^ 

3. has varied with the content being modeled 

• 4 , '"" • ' ■' y generally has been worth the tame . . 

■ ■ -X - ■ ■■■■ 

5. all time spent has been worthwhile 

20^. T.he new total system conceptualiz^tiori of -training developed by" 
TRADOC is the resu^it of several major intellectual activities, 
occurring' at TRADOC in the last few'years.' 'We are trying to 
determine how much influence the §,ofTech project has had on the 
emergence, of this new system. Please indicate'' yoiir estimate 
of SbfTech'^ contribution to the progress made by TFADOC in 
developing cAn integrated systems approach to training. 

1. ■ ' ' A- SofTech has not contributed 

■ • " ■ " . . ■ ■■ ', ■ ' 

2. SofTech 's contribution has been .minimal 

3. ■ SofTech made, a moderate contribution . ' 

4. SofTech was a major contributor 

5- SofTech was-- the most important single contributor 



21, Do you consider the tbtal system concept**of training as . elucidated 
by the Task 3 model an improvement over previous conc^eptualizations 
of training held by the Arnly? 

L. previous conceptualizations of training were* 

'superior 

2. ' provLous conceptualizations were, slightly better 

* 

3. ^ previous- (conceptualizations and total system concept 

are about equal ' ; 

4. .total system concept is slightly better 

^ 5. total system concept is superior 
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'l^lease respond to the following questiorts^'by wi^iting your answers in the 
space p}-ovided,> If yt>u needVmore space, continue „6n thje extra pages 
ikt the end of this booklet or\ atitj^ch an sh^&t*/ 
of the question^ coul3 be responded to with ^ brlBf comrient, I \Jould 
like to ask/ that you elaborate on yopr responses whenever possible. 

• • .»C ,- ■ ' * ' ' ' ' m ■■ • ■ ^ \ ■ , 

22, Qaestionnalire and interview :^data showed that the dpinions of TRADOC 
officers were more pqsitlve towards SABT at .t;tle end of Task 3 ' 
(May, 1977) than they were at the end of Task 2 (December, 1976), 
To what do ,you, at'tjribute this shift i^ a.ttitude?* 



3 



23. To .your knowledge, in what ways will the Task 3 raqdel be used by 
TRADOC? 
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24. It was anticipated that in addition to the SADT models produced/ the 
# TRADOC organization would improve their analytic and planning capa- 
bilities as a result of their participation in the project. 

In you opinion, has this project had an impact on TRADOC's ability 

solve large and/or complex problems? If so, how has TRADOC's 
analysis and planning capabilities improved? If you do not think 
the project has had an impact oil these abilities within TRADOC, why 
do you think this transfer of technical skills failed tb occur? 
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25. ' In what ways has TRADOC benefited from the SofTech project? 



\ 
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In what ways could the SofTech project have been altered to better 
meet TRADOC's needs? 



I' 
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' 27. Do you have any othef thoughts or comments about SADT or the SofTech 
project which you havq^ riot had an opportunity to express but believe 
to be important to this evaluation? 
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